Learning from Labeled Data: Supervised Techniques, Explained

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 7.1 Free with a free account

Learning from Labeled Data: Supervised Techniques

Introduction to Artificial Intelligence · Chapter 7: Machine Learning, NLP, and PerceptionSection 7.1 · lesson 49 of 6915 min · free with a free account

← Prolog: A Logic-Based Programming Language for AINext lesson: Discovering Patterns Without Labels: Unsupervised Techniques →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How models learn from labeled examples and how to test them honestly.

  • Distinguish classification from regression and explain how each is scored
  • Explain the roles of training, validation and test sets
  • Compare linear regression, logistic regression, nearest neighbors and naive Bayes
  • Use the bias-variance trade-off and cross-validation to choose a model

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (2,111 words). Read along, or search it.

Opening Question

Learning from 10,000 Past Loans

A bank holds 10,000 past loans. For every one, it knows what happened: the loan was repaid, or it wasn't. Those known outcomes act like an answer key. A program can study the examples, find what tends to go with repayment, and predict the outcome for a new applicant. So how does it learn from them? And how would the bank know it really learned, instead of just memorizing?

Definition

Features, Labels and Generalization

This is supervised learning: learning from examples whose correct answers, called labels, are provided. Each example has features, the inputs, such as an applicant's income, and a label, the answer to predict. A model is a rule with adjustable parameters. Training adjusts them until the predictions match the labels well. But the real goal is new examples the model has never seen. Doing well on those is called generalization.

Two Problem Types

Classification Versus Regression

Supervised problems come in two types, set by what the label is. In classification, the label is a category: spam or not spam, repaid or defaulted. The model picks a category, often with a probability. In regression, the label is a number: a price, or tomorrow's temperature. The model predicts a value. They're scored differently, too. Classification uses accuracy. Regression uses mean squared error: the average of the squared gaps between predictions and true values.

Data Splits

Training, Validation and Test Sets

Now, how would anyone know a model really learned? Scoring it on its own training examples proves nothing. A model can score perfectly there just by memorizing them, which is overfitting. So the labeled data is divided into three parts, each with its own job. The training set, usually 50 to 70 percent, fits the model's parameters. The model learns from this, and nothing else. The validation set, 15 to 25 percent, compares models and tunes settings. It may be used many times. And the test set, also 15 to 25 percent, gives one final, honest estimate. It's used once, at the very end.

Good Habits

Keeping the Split Honest

A few habits keep the split honest. Shuffle before splitting, because data often arrives in an order, by date or by customer, that makes the parts unlike each other. But for time-ordered data, split by time: predicting the future means the test set must come later than the training set. Keep rare categories in every part, or one might be missing from the test set altogether. Clean and scale the data after splitting. Doing it on all the data lets test information leak into training. And never tune on the test set. Each peek makes the final score a little more optimistic.

Linear Regression

Fitting a Line by Least Squares

Now the methods. Linear regression predicts a number, as a weighted sum of the features. With one feature, it's a straight line: price equals b zero, plus b one times size. b zero is the intercept, and b one is the slope. Here are six houses, from 40 to 120 square meters. Least squares picks the line that makes the sum of the squared errors as small as possible. For one feature there's a direct formula, and here it gives a slope of 2.16 and an intercept of 44.16. So each extra square meter adds about 2.16 thousand to the predicted price.

Worked Example

Errors and Mean Squared Error

Check the fit. The 40 square meter house is predicted at 130.5 thousand, so its error is minus 0.55. Down the table, every miss is under five thousand, either way. Square the six errors and average them, and the mean squared error is 12.16. For a new house of 90 square meters, the prediction is 238.5 thousand. With many features, the line becomes a flat surface, and least squares still finds the best fit.

Logistic Regression

Turning a Score into a Probability

Despite its name, logistic regression is a method for classification. It first computes a score, z, from the features, exactly as linear regression would. The score can be any number, so it's passed through the sigmoid function, which squeezes it into a probability between zero and one. If that probability is above one half, the model predicts the first category.

Worked Example

Study Hours and the Decision Boundary

Ten students record their study hours, and whether they passed. Fitting gives a score of z equals minus 4.84, plus 0.88 times hours. At one hour, the probability of passing is under two percent. At three hours, it's ten percent. At five and a half hours, the score is zero, and the probability is exactly fifty percent. That's the decision boundary. At eight hours, it's ninety percent, and at ten hours, ninety-eight. The parameters come from maximum likelihood: the computer improves them step by step, until the probabilities fit the labels best. With several features, the boundary is always a straight line, or a flat surface. That keeps the model simple and stable, but curved boundaries are out of reach.

Nearest Neighbors

Voting Among the k Closest Examples

k-nearest neighbors needs almost no training. It simply stores all the labeled examples. To predict for a new example, it finds the k stored examples closest to it. For classification, those neighbors vote. For regression, their labels are averaged. Two cautions. Features must be on comparable scales, or one large-valued feature will swamp the distance. And all the work happens at prediction time, so it becomes slow on very large data sets.

Choosing k

Small k Versus Large k

The setting k controls how smooth the boundary is. A small k follows every quirk of the data. A large k averages over a wide area. In one experiment, with 300 examples in two curved groups, k equals one scored a perfect 100 percent on its own training data. That tells us nothing, because every training example is its own nearest neighbor. On new data, measured by cross-validation, k equals one scored 89.7 percent, and k equals eleven did best, at 91.3. Very large k falls away again: 85.7 percent at k equals a hundred and one.

Naive Bayes

Multiplying Counted Chances

Naive Bayes uses Bayes' rule with one simplifying assumption: given the category, the features are independent of one another. The score for each category is its overall share, times the chance of each observed feature in that category. Then each score is divided by the total of all the scores, to give a probability. Training is just counting. Here are 12 past students: 6 passed, and 6 didn't. Of those who passed, five of six studied a lot. Of those who didn't, only one of six did. Poor attendance: two of six who passed, four of six who didn't. Short sleep shows the same split, two of six against four of six.

Worked Example

A 55.6 Percent Chance of Passing

A new student studies a lot, but has poor attendance and short sleep. The passed score is six twelfths, times five sixths, times two sixths, times two sixths: 0.0463. The did-not-pass score is six twelfths, times one sixth, times four sixths, times four sixths: 0.0370. Divide by their total, and the probability of passing is 55.6 percent. It's a close call, and the model says so honestly. Add one to each count, the usual fix for zero counts, and it becomes 51.9 percent. The weakness is the assumption itself. Study time and attendance move together, so they get counted twice, which can make the model overconfident.

Quick Check

The Mirror-Image Student

Quick check. Another student studies little, but has good attendance and enough sleep. Here are the same counts, read for those values. What does naive Bayes predict? Pause, and decide. Fail. The pass score is six twelfths, times one sixth, times four sixths, times four sixths: 0.0370. The fail score is six twelfths, times five sixths, times two sixths, times two sixths: 0.0463. So the chance of passing is 44.4 percent. It's the mirror image of the last student: every count changed sides, so the two scores swapped. A tie would need equal scores, and these aren't equal.

Comparison

Four Supervised Methods Side by Side

Here are the four methods, side by side. Linear regression predicts a number. It's fast and easy to interpret, but it can't capture curves unless features are added. Logistic regression gives a category with a probability. It's stable, but its boundary is always straight. Nearest neighbors learns nothing up front. It captures curves, but it's slow on big data, and needs scaled features. Naive Bayes is very fast and works with little data, but correlated features break its assumption.

Theory

Bias, Variance and Noise

Why do models that are too simple, and models that are too flexible, both fail? A model's expected error on new data has three parts. Expected error equals bias squared, plus variance, plus noise. Bias is error from wrong assumptions. A model that's too simple can't match the true pattern, even with unlimited data. That's underfitting. Variance is error from sensitivity to the particular training sample: a different sample gives a different model. Too much flexibility causes it, which is overfitting. Noise is randomness in the labels that no model can predict. It sets a floor on the error.

Experiment

Line, Cubic and Degree-Nine Fits

One experiment measures the first two. It draws 400 training sets of 30 points, all from the same noisy curve. A straight line has a bias squared of 0.20, and a tiny variance. Its total error is 0.30. A degree-nine curve has almost no bias, but a variance of 0.045. Its total is 0.135. Degree three balances the two, for the lowest total: 0.108. That's the bias-variance trade-off: making a model more flexible lowers bias, but raises variance.

Diagnosis

Reading Training and Validation Curves

In practice, watch two curves as a model gets more flexible. Training error always improves, so it's useless for choosing a model. Validation error falls, and then rises again. Its lowest point marks the best model. High error on both curves points to bias. A low training error, with a much higher validation error, points to variance.

Cross-Validation

Rotating Folds for Reliable Scores

A single validation set wastes data, and it can be unlucky. Cross-validation reuses the data more fully. Split the training data into k equal folds. Five or ten is typical. Train on all the folds but one, and validate on the one held out. Repeat until every fold has had a turn, then report the mean and the spread of the scores. So each example validates exactly once, and trains k minus one times. The cost is k times the training effort. Variants include stratified folds, which keep the category shares equal, and leave-one-out. And a gap smaller than the spread isn't convincing evidence.

Tuning

Hyperparameters and Grid Search

Parameters are numbers the training finds on its own, like the weights in a regression. Hyperparameters are settings you choose before training: k in nearest neighbors, a polynomial's degree, or the smoothing in naive Bayes. Tuning searches for the hyperparameter values that validate best. A grid search tries each candidate value, and scores it by cross-validation. Then it trains the winner on all the training data. The test set isn't touched until the very end. On the two-curve data, the best setting was k equals eleven, and its neighbors were almost as good, so the choice isn't delicate.

Comparison

Three Classifiers on Curved Data

Now compare the three classifiers on that data, where guessing would score about fifty percent. Logistic regression and naive Bayes both land near 83 percent. Nearest neighbors scores 91.3: a lead bigger than the spread between folds. And the untouched test set agrees. The groups are curved, so the flexible method suits this data. On other data the ranking could reverse. No method is best everywhere.

Quick Check

Why a Reused Test Score Is Optimistic

Quick check. A team tries thirty model variants, then reports the best one's accuracy on the test set, as its expected real-world accuracy. What's wrong with that number? Pause, and decide. It's too high, too optimistic. The test set was used to choose the model, so the winner was partly picked for its luck on those very examples. Choices belong on validation data, or in cross-validation. Then the test set is used once, at the end. The tries don't make the score too low. They push it the other way: with enough attempts, one looks good by luck.

Application

A Plan for the Bank's Loans

Back to the bank and its ten thousand loans. Repaid or not is a category, so this is classification. Set aside a test set first. Pick a simple baseline, then score a few candidates by cross-validation, and if several are within the spread, prefer the simplest. Train the winner on all the training data, and test it once. That one score answers the question: did it learn, or memorize?

Key Takeaways

Splits, Methods and Model Choice

To sum up. Supervised learning learns from labeled examples. Classification predicts a category, and regression a number. Fit on the training set, choose on validation data, and use the test set once. Linear regression fits a line, logistic regression turns a score into a probability, nearest neighbors vote, and naive Bayes multiplies counted chances. Error is bias squared plus variance plus noise, and cross-validation is a reliable guide for choosing a model.

Think About It

High Training, Low Validation Scores

Here's something to think about. A model scores 99 percent on its training data, but only 70 percent on the validation set. What's probably wrong, and what would you try first?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Learning from Labeled Data: Supervised Techniques

What is the difference between classification and regression?
Classification predicts a category, while regression predicts a number. Deciding whether an email is spam, or whether a loan was repaid or defaulted, is classification, and the model often gives a probability for each category. Predicting a house price or tomorrow's temperature is regression. Classification is usually scored by accuracy, and regression by mean squared error, the average of the squared gaps between predictions and true values.
Why do you need separate training, validation and test sets?
Separate sets keep the final score honest, because a model can score perfectly on its training data just by memorizing it. The training set, usually 50 to 70 percent, fits the parameters. The validation set, 15 to 25 percent, compares models and tunes settings. The test set, also 15 to 25 percent, is used once at the end, since choosing a model on it makes the score too optimistic.
What is the bias-variance trade-off?
The bias-variance trade-off is that making a model more flexible lowers bias but raises variance. Expected error equals bias squared plus variance plus noise. Bias comes from assumptions too simple to match the true pattern, causing underfitting, while variance comes from sensitivity to the particular training sample, causing overfitting. High error on both training and validation data points to bias; a large gap between them points to variance.
How does k-fold cross-validation work?
k-fold cross-validation splits the training data into k equal folds, often five or ten, trains on all folds but one, and validates on the one held out. It repeats until every fold has had a turn, then reports the mean and spread of the scores. Each example validates exactly once and trains k minus 1 times, at k times the training cost of a single split.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →