Measuring Data Quality: Precision, Accuracy, and Pitfalls

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 1.7 Free with a free account

Measuring Data Quality: Precision, Accuracy, and Pitfalls

Introduction to Artificial Intelligence · Chapter 1: Foundations of AI and Core ConceptsSection 1.7 · lesson 7 of 6915 min · free with a free account

← Core Ideas in Probability and ChanceNext lesson: Machine Learning vs. Artificial Intelligence: Key Differences →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

Accuracy versus precision, data quality, model metrics, and common data pitfalls.

  • Distinguish accuracy from precision in measurement and in model evaluation
  • Evaluate a dataset against the six dimensions of data quality
  • Calculate accuracy, precision, recall, and F1 from a confusion matrix and match the metric to the cost of errors
  • Recognize sampling bias, imbalanced data, data leakage, overfitting, and confusing correlation with causation

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (2,020 words). Read along, or search it.

Opening Story

99 Percent Accurate Yet Caught No One

A health-screening app is tested on 1,000 patients, and the team proudly reports 99% accuracy. Then a doctor asks a simple question: how many of the sick patients did it catch? None. Only 10 of the 1,000 were sick, and the app simply said healthy every time. One impressive number hid a useless model. So how do we measure data and results honestly, and spot the traps that flatter them?

Two Qualities

Closeness to Truth Versus Consistency

People often use accurate and precise as if they meant the same thing. In measurement, they're two different qualities. Accuracy is how close a measurement is to the true value. Precision is how close repeated measurements are to each other: their consistency.

Worked Example

Four Scales Weighing 70 Kilograms

Picture weighing someone whose true weight is 70 kilograms, on four scales, three readings each. Scale A reads 70.0, 70.1 and 69.9. Close to 70, and close to each other: accurate and precise. Scale B reads 67.5, 72.4 and 69.8. Right on average, but jumpy: accurate, not precise. Scale C reads 74.1, 74.0 and 74.2. Very consistent, but about 4 kilograms too high: precise, not accurate. And scale D, at 66.0, 75.3 and 72.1, is neither. So a tool can be precise without being accurate, like a scale that always reads 4 kilograms high.

Two Warnings

Extra Digits and Shifting Meanings

Two warnings. First, extra digits aren't accuracy. Writing a reading as 70.12345 kilograms looks exact. But a scale that rounds to the nearest kilogram can't truthfully support five decimal places. Second, the words change meaning when we judge a model. There, accuracy is the share of predictions that were correct, and precision is how often a yes is right. The ideas are related, but not identical, so always check which meaning is in use.

Messy Data

Five Problems in a Customer List

Good data takes more than careful measurement. Look at this small customer list. Chen Wu's age is 212. Dina Khan's email is blank. Ben Ortiz appears twice, with the same ID. Eli Novak's sign-up date is written month first, unlike the others. And Farah Ali's country says USA, where other rows say United States. Each of these problems fails a different test of quality.

Six Dimensions

How Data Quality Is Judged

Data quality is judged along six dimensions. Each asks its own question, and a dataset can pass some while failing others. Accuracy: do values match reality? An age of 212 doesn't. Completeness: is anything important missing, like that email? Consistency: is the same thing written the same way? USA and United States aren't. Validity: do values follow the rules? That date breaks the year-month-day format. Timeliness: is the data recent enough? Last decade's prices won't predict this month's sales. And uniqueness: is each real thing recorded once? Ben is listed twice. A seventh question is whether the data represents the people the model will meet. That's sampling bias, coming up shortly.

Confusion Matrix

Four Boxes for Yes-or-No Predictions

Many AI models make yes-or-no calls: spam or not, sick or healthy, fraud or legitimate. The thing a model hunts for is called the positive case. Say a spam filter is tested on 100 emails, and 20 really are spam. Every prediction lands in one of four boxes. True positive: it said spam, and it was. Fifteen of those. False positive: it said spam, but the email was good. A false alarm. Four. False negative: it said good, but it was spam. A miss. Five. And true negative: it said good, and it was. Seventy-six. The four counts make a confusion matrix.

Four Metrics

Accuracy, Precision, Recall, and F1

The same four counts give four different scores, and each answers its own question. Accuracy: out of all the emails, how many were handled correctly? Fifteen plus seventy-six, out of a hundred: ninety-one percent. Precision: when the filter says spam, how often is it right? Fifteen out of nineteen: about seventy-nine percent. Recall: out of all the real spam, how much did it catch? Fifteen out of twenty: seventy-five percent. F one balances the two: two times fifteen, over two times fifteen, plus four, plus five. That's thirty out of thirty-nine, about seventy-seven percent. A memory aid. Precision is about the alarms: can you trust the flags? Recall is about the catch: how much of the real spam was found?

Quick Check

Recall for Bridge Crack Detection

Quick check. A drone photographs a bridge, and a model scans 200 photos for cracks. The numbers are illustrative. Twenty-five photos really show cracks. The model flags thirty photos, and twenty of those really show cracks. Using these counts, what is the model's recall? Pause, and decide. Twenty out of twenty-five: eighty percent. Recall starts from the real cracks, all twenty-five, and asks how many the model found. It found twenty. Twenty out of thirty is the tempting choice, but that's precision: it starts from the flags. Here, two of every three flags are real.

Why F1

The Harmonic Mean Exposes Weakness

Why not just average precision and recall? Because an average can hide a serious weakness. F one uses a stricter average, the harmonic mean, which stays low whenever either score is low. Precision eighty percent, recall eighty percent: both methods say eighty. But precision one hundred percent, with recall ten percent? The simple average says fifty-five. F one says about eighteen. So a model can't earn a high F one by being excellent at one score, and terrible at the other.

Threshold Tradeoff

Choosing Metrics by the Cost of Mistakes

Most models give each case a score, and a threshold decides when the score is high enough to say yes. Raise the threshold, and the filter gets stricter. It blocks fewer good emails, so precision tends to rise. But more spam slips through, so recall falls. Lower it, and the opposite happens. No setting is best for every purpose. So choose by the cost of mistakes. Screening for a serious disease? Missing a sick patient is worst, so watch recall. Blocking email that may be important? Wrongly blocking a real email hurts most, so watch precision. Spotting fraud, where both mistakes are costly? Watch F one, or both scores together. And with balanced data and equal costs, accuracy is reasonable.

Five Pitfalls

Traps That Flatter Results

Even with good metrics, results can mislead. Five pitfalls account for a large share of disappointing AI projects. Sampling bias. Imbalanced data. Data leakage. Overfitting and underfitting. And mistaking correlation for causation.

Sampling Bias

The 1936 Literary Digest Poll

Sampling bias means the data isn't a fair picture of the situation the model will face. In 1936, the Literary Digest magazine mailed about ten million ballots, to names from its subscriber lists, phone books and car registrations. About 2.4 million came back. It predicted that Alf Landon would win. Franklin Roosevelt won in a landslide. In the Depression, those lists leaned wealthier than voters as a whole. And the people who chose to reply weren't typical either. Modern AI repeats the mistake. A voice assistant trained mostly on one accent understands others poorly. A health model built from smartwatch wearers may not fit everyone else. The fix: compare the sample's make-up with the real population, gather data from every group the model will serve, and be honest about who's left out.

Imbalanced Data

When the Easy Answer Wins

Back to the screening app from the start. Nine hundred ninety patients were healthy, and ten were sick. A model that always says healthy is right nine hundred ninety times out of a thousand. That's ninety-nine percent accuracy, and a recall of zero. When one outcome is rare, accuracy mostly measures how common the easy answer is. So report recall, precision and F one for the rare outcome. Compare the model with a baseline that always guesses the common answer. Any useful model must beat it. And gather more rare cases, or give them more weight in training.

Data Leakage

Seeing the Answer Key Early

Data leakage: a model is given information it won't have, at the moment it must predict. It's like a student who sees the answer key during practice. The practice score is excellent, and the real exam goes badly. It takes three common forms. First, an input that reveals the outcome, like a follow-up emergency visit, used to predict whether a patient returns to hospital. Second, test data used during preparation, such as rescaling all the data before setting the test records aside. Third, the same item on both sides: one customer in both the training and the test records. For every input, ask: would I know this when I need the prediction?

Overfitting and Underfitting

Memorizing Versus Skimming

Picture two students preparing for an exam. One memorizes last year's exact questions and answers, and is lost when the questions change. The other only skims the headings, and can't answer detailed questions. The first is overfitting: learning the training data too closely, noise included, so it fails on new data. The second is underfitting: too simple to capture the real pattern. Held-out test data tells them apart: records set aside, and never used for training. Underfitting scores low on both. A good fit scores high on both, and close together. Overfitting scores very high in training, and much lower on the test. The remedies: a richer model for underfitting. For overfitting, a simpler model, more data, or fewer noisy inputs.

Correlation and Causation

Firefighters, Fire Size, and Confounders

Two things are correlated when they tend to rise and fall together. One causes the other only if changing the first really changes the second. Fire records show that bigger fires have more firefighters, and also more damage. Would sending fewer firefighters cut the damage? No. A hidden third factor, the size of the fire, drives both. A hidden factor like that is called a confounder. So a pattern has four possible explanations: A causes B, B causes A, a hidden factor causes both, or plain coincidence. Models learn correlations. Proving cause usually needs a controlled experiment, so be careful turning a model's pattern into a claim about why.

Quick Check

A Refund Field Predicting Returns

Quick check. An online store builds a model to predict which orders will be returned. Its strongest input is a field called refund issued. On held-out test orders, it finds 98% of returns, with very few false alarms. What's the most likely problem with this model? Pause, and decide. Data leakage. A refund is issued after an order comes back, so the field records the outcome itself. When a new order is placed, nobody knows it yet. Overfitting is tempting, because the score looks too good. But overfitting shows a gap between training and test, and this score is high on the test itself. And imbalance can't explain it: the model finds nearly every return. Drop the field, and measure again.

Data Audit Checklist

Inspect the Ingredients Before Cooking

Most failures were visible in the data before any model was built. Think of inspecting ingredients before cooking: no recipe rescues spoiled food. Know the question, and what each mistake costs. Trace where the data came from: who collected it, where, and when. Count the outcomes, to spot a rare one. Scan for missing, invalid, duplicate and inconsistent values. Look at the spread, for impossible values and sudden spikes. Test every input for leakage. Split off the test data first, before any preparation. And set a baseline, with metrics that match the costs.

Worked Example

Auditing a Cancellation Model

Here's the checklist at work. A company wants to predict which of its 10,000 customers will cancel. A quick audit finds four problems. Only 3% cancel, so always predicting stays scores 97% accuracy. That's imbalance: report recall, precision and F one, and beat that baseline. 120 customers appear twice. That's a uniqueness problem, and it could put one customer on both sides of the split, so remove duplicates before splitting. A field recording the date a customer quit is filled in only for those who left. That's leakage: drop it. And every record came from mobile-app sign-ups. That's sampling bias: collect web customers too, or state the limit clearly.

Key Takeaways

Quality, Metrics, and Pitfalls

To sum up. Accuracy is closeness to the truth, and precision is consistency. A tool can have either one without the other. Data quality has six dimensions: accuracy, completeness, consistency, validity, timeliness and uniqueness. A confusion matrix gives accuracy, precision, recall and F one. Pick the metric that matches the cost of mistakes. When one outcome is rare, high accuracy can hide a model that finds nothing. And watch for sampling bias, leakage and overfitting, and remember that correlation isn't causation.

Reflection

Choosing Between Two Fraud Models

Here's something to think about. One fraud model catches 90% of fraud, but blocks many honest customers. Another blocks very few honest customers, but catches only half the fraud. Which would you choose, and what would change your mind?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Measuring Data Quality: Precision, Accuracy, and Pitfalls

What is the difference between accuracy and precision in measurement?
Accuracy is how close a measurement is to the true value, while precision is how close repeated measurements are to each other. A scale reading 74.1, 74.0, and 74.2 kilograms for a person who truly weighs 70 is precise but not accurate. Extra decimal places do not add accuracy either, since a scale that rounds to the nearest kilogram cannot support a reading like 70.12345.
How do you calculate precision and recall from a confusion matrix?
Precision is true positives divided by all predicted positives, and recall is true positives divided by all actual positives. For a spam filter with 15 true positives, 4 false positives, and 5 false negatives, precision is 15 out of 19, about 79 percent, and recall is 15 out of 20, or 75 percent. F1 combines them as 30 out of 39, about 77 percent.
Why can high accuracy be misleading on imbalanced data?
When one outcome is rare, a model can score high accuracy just by predicting the common answer every time. If 10 of 1,000 patients are sick, a model that always says healthy is 99 percent accurate but has a recall of zero, catching no sick patients. Report precision, recall, and F1 for the rare outcome, and check that the model beats a baseline that always guesses the common answer.
What is data leakage in machine learning?
Data leakage happens when a model is given information it will not have at the moment it must make a real prediction. Common forms include an input that reveals the outcome, such as a refund issued field used to predict product returns, test data used during preparation, and the same customer appearing in both training and test records. Leakage produces excellent test scores that fall apart in real use.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →