Teaching Machines to Recognize and Sort Images, Explained

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 8.2 Free with a free account

Teaching Machines to Recognize and Sort Images

Introduction to Artificial Intelligence · Chapter 8: Perception, Robotics, and Hands-On AISection 8.2 · lesson 58 of 6917 min · free with a free account

← How Machines Understand Spoken LanguageNext lesson: Lisp: The Language That Shaped Early AI →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How computers classify, detect and segment objects in digital images.

  • Describe how images are stored as pixels and processed through preprocessing, features and a classifier
  • Distinguish classification, object detection, and semantic and instance segmentation
  • Explain how data splits, augmentation and transfer learning affect image model performance
  • Calculate precision, recall, F1 and intersection over union to evaluate image models

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (2,230 words). Read along, or search it.

Opening Question

Checking Bottles on a Conveyor Belt

A factory camera photographs every bottle on a conveyor belt. Within a fraction of a second, a program must decide: is this bottle fine? Where does the label sit? And which pixels show a crack? People do this without effort. For decades, it was one of the hardest problems in AI. Because a photograph is only a grid of numbers, and nothing in those numbers says bottle. So how does a program get from numbers, to a decision?

What an Image Is

Grids of Pixels and Color Channels

Start with what an image is. A digital image is a grid of pixels. In a grayscale image, each pixel is one number, for brightness. Zero is black, and larger numbers are brighter. In a color image, each pixel has three numbers: red, green and blue. Those are the three channels, so a color photo is height, by width, by three. A tiny handwritten digit, eight by eight, is just 64 numbers. A small color photo of 32 by 32 is 3,072. A phone photo is about 36 million.

Why It Is Hard

Same Object, Very Different Numbers

Why is recognition hard? Because the same object can produce wildly different numbers. Lighting: a face is bright at noon, and dark at dusk. Viewpoint and scale: a mug from above looks nothing like one from the side. Position: shift the object a few pixels, and every number changes. Occlusion and clutter: part of it is hidden, or the background is busy. And variation within a category: cats differ in color, size and pose, yet all of them are cat. A model must tolerate all of this, and still tell different things apart.

Classic Pipeline

Preprocess, Measure Features, Classify

A classic image pipeline has three steps. Preprocessing makes images comparable. Features measure what helps tell categories apart. And a classifier turns those measurements into a label. Preprocessing comes first, because a model expects every input to have the same size, and a similar range of values. So images are resized and cropped, brightness is normalized, color is dropped if it doesn't matter, noise is smoothed, and faces are aligned upright.

Experiment Results

Dimmed Digits and a One-Line Fix

Here's why preprocessing matters. A classifier learned eight-by-eight digits from 1,257 images, and was tested on 540 new ones. At normal brightness, it scored 98.1 percent. Then the test images were simply dimmed. At half brightness, accuracy fell to 90 percent. At 30 percent brightness, it fell to 56.9. The fix was one line of preprocessing: rescale each image so its brightest pixel equals one. Then all three scored 98.1. A model sees only numbers. Whatever preprocessing doesn't standardize becomes a trap, so run the same steps on every new image.

Features

From Raw Pixels to Learned Features

Step two is features. A feature is a measurement that helps tell categories apart. Raw pixels are the simplest features, but they change drastically with a small shift. They work for small, centered images, like the digits. A color histogram counts pixels in each color range, which helps tell ripe fruit from unripe. Edges and gradients capture shapes and outlines. HOG counts edge directions in small cells, and is good for pedestrians. Keypoints are distinctive corners and blobs, for matching objects across photos. And most modern systems use learned features, patterns a neural network discovers from data.

Edge Detection

Sobel Filters and Smoothing

Take edges. An edge is a place where brightness changes quickly. A Sobel filter estimates how fast brightness changes from left to right, and from top to bottom, at each pixel, and combines the two into an edge strength. But noise creates false edges everywhere. Smooth the image first, and the true outlines stand out cleanly.

Classifiers

Test Accuracy Is the Honest Number

Step three: a classifier maps the features to a label, using a method such as logistic regression, nearest neighbors, a support vector machine, or a neural network. On the same 540 test digits, logistic regression scored 97.2 percent, k-nearest neighbors 98.0, and the support vector machine 98.1. Each scored higher on its own training images, above 99 percent. Models always look better on data they've seen, so test accuracy is the honest number.

Three Tasks

Classification, Detection and Segmentation

The same image can be asked three kinds of question, each with a richer answer. Classification asks: what is in this image? It gives one label for the whole image, or several, with a confidence. Good bottle, or defective bottle. Object detection asks: what is where? For every object it finds, it gives a box, a category, and a confidence. Finding faces, or locating the label on a bottle. Segmentation asks: which pixels belong to which thing? It labels every pixel, giving a mask. Outlining a tumor, or measuring a crack.

Task Details

Semantic Versus Instance Segmentation

Some details matter. Classification can be single-label, exactly one category, or multi-label, where several apply. Detection can report many objects in one image, so a model must decide when two overlapping boxes describe the same object. Segmentation comes in two forms. Semantic segmentation gives every pixel a category, but treats two people standing together as one region. Instance segmentation gives each object its own mask. And richer outputs need richer labels. A label per image is cheap, a box per object costs more, and a pixel-accurate outline costs the most.

Quick Check

Counting Cars in a Parking Lot

Quick check. A parking app looks at photos of a parking lot, and must report how many cars are in each one. Some cars are parked bumper to bumper. Which task fits the job: classification, object detection, or semantic segmentation? Pause, and decide. Object detection. It returns one box per car, and counting the boxes counts the cars. Classification only says cars are present. Semantic segmentation marks every car pixel, but it doesn't separate cars that touch, so it can't count them.

Datasets

MNIST, CIFAR-10, ImageNet and COCO

Image models learn from labeled examples, so the dataset largely decides what a model can do. A few public datasets have driven progress. MNIST holds 70,000 handwritten digits. CIFAR-10 holds 60,000 small color photos in 10 categories. The ImageNet challenge uses about 1.28 million training photos in 1,000 categories. COCO has about 330,000 photos, with 80 object categories.

Data Splits

Training, Validation and Test Sets

To judge a model honestly, the data is split three ways. Training fits the model. Validation compares settings while you build. And the test set is used once, at the end, for the honest score. For the digits: 1,257 images for training, about seventy percent, and 540 for testing, about thirty. Validation is carved out of the training part.

Common Pitfalls

Label Errors, Leakage and Dataset Shift

Datasets go wrong in predictable ways. Label errors: a cat photo marked dog. Unrepresentative data: faces mostly from one region, or photos only in daylight. Category imbalance: one defective bottle in a hundred. Leakage: near-copies of training images end up in the test set, so the score is inflated. Dataset shift: a model trained on sharp studio photos meets blurry phone shots. And privacy and licensing: images collected without consent. A practical rule: split the data before any augmentation or tuning, and split by source, by patient or by scene, so near-duplicates never straddle the boundary.

Augmentation

Changes That Keep the Label

When labeled images are scarce, augmentation helps. It makes extra training images with changes that shouldn't alter the label: a small shift or rotation, a zoom, brightness, or noise. That teaches the model those changes don't matter. But some changes do alter the answer. A large rotation can turn a six into a nine. Mirroring suits animals and street scenes, but not digits or letters. And changing colors hurts when color is the signal. The key question is always: could this change alter the correct answer?

Experiment Results

One-Pixel Shifts and Shifted Copies

An experiment shows how much this matters. A model trained on the original digits scored 98.1 percent. Moving the test images by just one pixel dropped it to between 13 and 47 percent, depending on the direction. Retrained with nine shifted copies of every image, it scored about 98 to 99 percent on shifted images, and 99.1 on unshifted ones. In an eight-pixel image, one pixel is a big move. Real photos are less fragile, but the same effect appears.

Transfer Learning

Feature Extraction and Fine-Tuning

Training a deep image model from scratch can need millions of labeled images. Transfer learning avoids that. Start from a model trained on a huge dataset, like ImageNet. Its early layers detect edges, textures and shapes, useful for nearly any image task. Keep that knowledge, and teach only the final decision. Feature extraction freezes the pretrained layers, and trains a new classifier on top: fast and safe with very little data. Fine-tuning updates some or all layers, in small steps, when you have more data. Training from scratch suits huge datasets, or very unusual images.

Experiment Results

Pretrained ResNet-18 Features on CIFAR-10

In one experiment, a standard network called ResNet-18, pretrained on ImageNet and frozen, supplied features for photos from CIFAR-10. Chance is ten percent. With only five labeled photos per category, the pretrained features reached about 61 percent. Raw pixels reached 22. Even with 100 photos per category, raw pixels managed only 28, against 78. The pretrained network already knew how to see. Some cautions. X-rays are far from everyday photos, so the gain shrinks. Match the model's input size and normalization. It inherits its data's gaps and license. And still test on held-out images.

Confusion Matrix

Where the Mistakes Fall

Now, measuring a model. For classification, start with a confusion matrix: for each true category, count how many images were assigned each predicted category. Correct answers fall on the diagonal. Mistakes fall elsewhere, and show which categories get confused. The digit classifier got 530 of 540 right. A four was called an eight twice, and a six was called a four twice. Looking at exactly which images were confused is called error analysis, and it guides what to fix.

Performance Measures

Accuracy, Precision, Recall and F1

Four numbers summarize performance. Accuracy is the share of all images labeled correctly. Precision, for one category, is true positives over everything flagged as that category. Of the images flagged, what share truly are? Recall is true positives over everything truly in the category. Of the real cases, what share were found? And F1 combines them: two times precision times recall, over precision plus recall.

Worked Example

Why Accuracy Misleads on Rare Defects

Here's why accuracy alone misleads. A camera system checks 1,000 bottles, and 10 of them are defective. It catches 8 of the defects and misses 2. It also flags 15 good bottles, and clears the other 975. Accuracy is 98.3 percent. Precision is 8 out of 23 flagged, about 0.35. Recall is 8 out of 10, 0.80. F1 is about 0.48. Now a lazy program that calls every bottle fine scores 99 percent accuracy, and finds no defects at all. When one category is rare, accuracy is a poor guide.

Quick Check

Precision and Recall for Damaged Goods

Quick check. A model looks for damaged goods in a set of photos. It flags 40 photos as damaged, and 30 of those truly are. The whole set contains 50 damaged photos. What are its precision and recall? Pause, and work it out. Precision is 30 out of the 40 flagged: 0.75. Recall is 30 out of the 50 that were damaged: 0.60. Three quarters of its flags are right, but it found only sixty percent of the damage. Don't swap them: precision divides by what was flagged, recall by what was really there.

Thresholds

Trading Recall Against Precision

Most classifiers produce a confidence score, and a threshold turns it into yes or no. Lowering the threshold finds more real cases, so recall rises, but it raises more false alarms, so precision falls. Take a detector for the digit eight. At threshold 0.1, precision is 0.35 and recall 0.94. At 0.5, precision is perfect, and recall is 0.21. The average precision sums up every threshold in one score: 0.83 here. But picking the threshold is a choice about costs, and the model can't make it.

Detection Measure

Intersection Over Union for Boxes

Detection needs the right place too. Intersection over union, or IoU, measures how well a predicted box matches the true box. The model's box runs from 10 to 50 on each side, and the true box from 20 to 60. Each covers 1,600 square units. They overlap in a 30 by 30 square: 900. Together they cover 1,600 plus 1,600, minus the overlap: 2,300. So IoU is 900 over 2,300, about 0.39. A detection usually counts only at 0.5 or more, so this close-looking box is a miss. A box just four units right and two down from the true one scores 0.75: a hit. Over all detections, the scores combine into mean average precision, the standard summary.

Segmentation Measures

Mask Overlap, Dice and Robustness

Segmentation uses the same idea on pixels: the overlap of the predicted and true masks, over their union. The Dice score is similar: twice the overlap, divided by the sum of the two mask sizes. Plain pixel accuracy is a poor measure here. If the object is small, a model that marks everything as background still gets most pixels right. And one number is never the whole story. Test robustness on dimmer or blurrier images. Report results by group, like skin tone or camera. Check that confidence matches how often the model is right. And mind speed, memory and cost.

Key Takeaways

Image Recognition in Summary

To sum up. Images are grids of numbers, and a classic pipeline preprocesses, measures features, and classifies. Classification labels the whole image, detection finds objects with boxes, and segmentation labels every pixel. Split data honestly by source, use augmentation that keeps the label, and borrow a pretrained model when labels are few. And when a category is rare, judge with precision, recall and F1, and with IoU for boxes and masks, not accuracy alone.

Think About It

Testing a Skin-Check App

Here's something to think about. A skin-check app was trained on 20,000 photos, mostly of light skin, and it scores 95 percent on a test set from the same collection. Which dataset problems worry you, and how would you test the app before it's released?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Teaching Machines to Recognize and Sort Images

What is the difference between image classification, object detection, and segmentation?
Classification gives one or more labels for a whole image, object detection finds each object and returns a box, a category and a confidence, and segmentation labels every pixel to produce a mask. Counting touching cars calls for detection, because each car gets its own box. Semantic segmentation gives each pixel a category but merges touching objects of the same kind, while instance segmentation gives each object its own mask.
How do you calculate intersection over union for a bounding box?
Intersection over union, or IoU, is the area where the predicted and true boxes overlap divided by the total area they cover together. For two 40 by 40 boxes, one running from 10 to 50 on each side and one from 20 to 60, the overlap is 900 and the union is 1,600 plus 1,600 minus 900, or 2,300, so IoU is about 0.39. A detection commonly needs 0.5 or more to count.
Why is accuracy misleading for imbalanced image datasets?
Accuracy can look excellent while a model misses the rare category entirely. If 10 of 1,000 bottles are defective, a program that calls every bottle fine scores 99 percent accuracy yet finds no defects. Precision, the share of flagged items that are truly positive, and recall, the share of real positives that were found, reveal the problem. F1 combines them as two times precision times recall, divided by precision plus recall.
What is transfer learning in computer vision?
Transfer learning reuses a model trained on a huge dataset, such as ImageNet, so a new image task needs far fewer labeled examples. Its early layers already detect edges, textures and shapes. Feature extraction freezes those layers and trains a new classifier on top, which suits very little data. Fine-tuning updates some or all layers in small steps when more data is available. The gain shrinks for images far from everyday photos, such as X-rays.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →