Handling Uncertainty and Measurement in Scientific Work

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 4.1 Free with a free account

Handling Uncertainty and Measurement in Scientific Work

Introduction to Artificial Intelligence · Chapter 4: Search, Heuristics, and Game StrategySection 4.1 · lesson 23 of 6915 min · free with a free account

← Real-World Uses of Machine LearningNext lesson: Applying Bayes' Rule to AI Decisions →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

Measure, quantify, and report uncertainty in experiments and AI model results.

  • Distinguish random from systematic error and choose the right remedy for each
  • Calculate a standard deviation, standard error, and 95 percent confidence interval
  • Propagate independent uncertainties through sums, differences, products, and quotients
  • Report results with correctly rounded uncertainty and judge whether two model scores truly differ

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,989 words). Read along, or search it.

Opening Problem

Is 91.4 Percent Really Better

Two students measure the same table with a meter stick. One reads 150.2 centimeters, and the other reads 149.8. Neither was careless, and neither is exactly right. Every measurement carries some uncertainty. AI work has the same problem. Model A scores 91.0 percent on a test, and model B scores 91.4. Is B really better, or is that gap just the normal wobble of testing? Without a statement of uncertainty, nobody can tell. This lesson gives you the tools to tell.

Definition

Random Error Versus Systematic Error

The difference between a measurement and the true value is its error. Errors come in two kinds, and they behave very differently. Random error is unpredictable variation from one reading to the next. The readings scatter above and below the true value. Systematic error is a consistent push in the same direction, so every reading is off by about the same amount. Think of darts. Random error is a loose cluster around the bullseye. Systematic error is a tight cluster, in the wrong place.

Comparison

Different Causes, Different Remedies

They differ in every way that matters. Random error goes sometimes too high and sometimes too low. Systematic error always goes the same way. Random error comes from noise, or reading a scale by eye. Systematic error comes from a scale that isn't zeroed, a miscalibrated sensor, or a biased sample. Averaging many readings shrinks random error. Systematic error stays, however many readings you average. And you can't see systematic error in the scatter. All the readings agree, and all of them are wrong. So the remedies differ. For random error, repeat and average. For systematic error, calibrate against a known reference, or fix the method or the sampling.

Illustration

Averaging Cannot Remove a Fixed Offset

Here's what averaging does, in numbers. These values are illustrative. With random error alone, the typical error of the average keeps shrinking as readings pile up. Now add a fixed offset. The error of the average still falls at first, but it stops improving once it reaches the size of the offset. More readings can't remove it. Repeating is a powerful remedy for random error, and no remedy at all for systematic error.

Classifying Sources

Sorting Rulers, Scales, Runs, and Blunders

Real sources sort cleanly. Reading a ruler from a slightly different angle each time is random: it errs both ways. A scale that shows half a kilogram with nothing on it is systematic: every weighing comes out too high. Ten training runs, with different random starting settings, scoring 90.1 to 91.3 percent, is random: chance varies each run. A survey of only app users, used to describe all customers, is systematic. Repeat it, and it leans the same way. And a blunder is a third thing: a plain mistake, like typing 21.3 as 213. Careful checking, and a look at readings far from the rest, will catch most of them.

Quick Check

Timing Jitter and a Late Timer

Quick check. You time one program 100 times on a laptop. Background tasks make some runs a little slower, and some a little faster. And your timer always starts 0.05 seconds late. You average all 100 timings. Which error does the averaging remove? Pause, and decide. Only the jitter. Background tasks push runs both ways, so that's random error, and it averages out. The late start pushes every run the same way. A thousand timings would all still be off by the same amount. That needs a fix, not more runs: check the timer against a known reference. Both is the tempting answer. But averaging never touches a constant offset.

Worked Example

Standard Deviation of Five Rod Readings

Now the numbers. When a measurement is repeated, the mean, the ordinary average, is the best single estimate of the true value. Here are five readings of a metal rod: 12.1, 12.3, 12.0, 12.4, and 12.2 centimeters. Their mean is 12.2. The standard deviation, written s, says how spread out the single readings are: roughly, how far a typical reading sits from the mean. Step one: subtract the mean from each reading. Step two: square each difference. The squares add up to 0.10. Step three: divide by one less than the number of readings. 0.10 over 4 is 0.025. Step four: take the square root. s is about 0.16 centimeters. Why one less? The mean came from these same readings, so they look a little less spread out than they really are.

Rule of Thumb

68 and 95 Percent for Bell Shapes

For bell-shaped data, there's a handy rule of thumb. About 68 percent of readings fall within one standard deviation of the mean, and about 95 percent within two.

Standard Error

Four Times the Readings Halves Uncertainty

But what we usually report is the mean, and the mean is more reliable than any single reading. Its uncertainty is the standard error. The standard error is s divided by the square root of n. For the rod, 0.16 over the square root of 5 is about 0.07 centimeters. Don't confuse the two. The standard deviation is the spread of single readings, and it settles as you take more. The standard error is the uncertainty of the mean, and it keeps shrinking. Because of the square root, progress is slow. With s at 0.16, five readings give a standard error of 0.072 centimeters. Twenty readings give 0.036. Eighty give 0.018. Three hundred twenty give 0.009. Each row needs four times as many readings as the one above, and buys only half the uncertainty. To halve the uncertainty of a mean, take four times the readings.

Confidence Intervals

What a 95 Percent Interval Means

A single mean hides how uncertain it is. A confidence interval turns the standard error into a range. A 95 percent confidence interval is built by a method, one that captures the true value in about 95 of every 100 repeats of the experiment. It's the mean, plus or minus a multiplier times the standard error. With few readings the multiplier is larger: 2.78 for five readings. With many, it approaches 2. For the rod: 12.2, plus or minus 2.78 times 0.07. That's 12.2 plus or minus 0.2, so the interval runs from about 12.0 to 12.4 centimeters. Here's what 95 percent means. Picture the whole five-reading experiment run over and over, each time building its own interval. Most intervals contain the true length. Now and then, one misses it. That's expected. The method promises about 95 percent, not 100.

AI Application

Uncertainty in Test Accuracy

Test accuracy is an average too, so it has uncertainty. Its standard error is the square root of p times one minus p, over n. Back to model A and model B, each tested on 500 cases. Model A's 91.0 percent has a standard error of 1.28 points. So its 95 percent interval runs from about 88.5 to 93.5. Model B's 91.4 percent gives about 88.9 to 93.9. The intervals overlap almost completely. This test gives no good reason to call B better. That would need more test cases, or a bigger gap.

Propagation

Combining Uncertainties in a Robot's Speed

Often we don't measure the quantity we want directly. We measure other things, and calculate. Then each value's uncertainty propagates into the answer. For independent uncertainties, square them, add, and take the square root, like the sides of a right triangle. Errors partly cancel, so that's less than plain adding. For a sum or a difference, combine the absolute uncertainties that way. For a product or a quotient, combine the relative uncertainties, then multiply by the result. Multiplying by an exact constant multiplies the uncertainty by that constant. And raising to a power multiplies the relative uncertainty by the power. Here's a quotient. A robot travels 2.00 meters, plus or minus 0.02, in 4.0 seconds, plus or minus 0.2. Its speed is 0.50 meters per second. The relative uncertainties are 1 percent for distance, and 5 percent for time. Combined, that's 5.1 percent, so the speed is 0.50 plus or minus 0.03 meters per second. Notice that 5.1 is barely more than the 5 percent from time alone. The biggest relative uncertainty dominates. Improve that measurement first; polishing the distance would be wasted effort. These rules assume random, unrelated errors. A systematic error that pushes several measurements the same way doesn't combine like this, and needs separate care.

Repeatability

Repeatability and Reproducibility

An experiment is only trustworthy if its results can be obtained again. Repeatability: the same person, method and equipment get similar results on a repeat. Reproducibility: different people or equipment get similar results with the same method. Another team rebuilds your model from its description, and gets a similar score. Repeating serves three purposes. It estimates the random error, it exposes blunders and unstable results, and it makes the mean more reliable.

Sampling

Random Samples and Sample Size

Often we can't measure everything. Nobody can test a model on every email that will ever exist. So we measure a sample, and use it to learn about the whole population. Two things matter. First, how the sample is chosen. A random sample gives every member of the population an equal chance, which guards against the systematic error of a biased sample. Second, how big it is. Draw many samples of ten, and their averages spread widely. Draw samples of a hundred, and their averages cluster much more tightly around the truth, just as the standard error predicts.

Best Practices

Five Habits for Trustworthy Experiments

Five habits keep experiments trustworthy. Repeat runs, with different random starting values. And choose samples at random: shuffle the data before splitting it. Change one thing at a time, so you know what caused what. Compare against a baseline, such as the previous model. And record everything: settings, software and data versions, and random starting values, so others can repeat your work.

Reporting

Value, Uncertainty, Unit, and Rounding

A result without its uncertainty is incomplete, and it can mislead. The standard form is the value, plus or minus the uncertainty, and the unit. Then a short note: what kind of uncertainty it is, such as a standard deviation or a 95 percent interval, and how many measurements it comes from. Round the uncertainty to one significant figure. Two are allowed if its first digit is 1. Then round the value to the same decimal place. So 12.2034 plus or minus 0.1581 centimeters shows digits that mean nothing. Written properly, it's 12.20 plus or minus 0.16.

Quick Check

Rounding a Mean Run Time

Quick check. A program's mean run time is 3.4172 seconds, with a standard error of 0.0386 seconds. Which way of writing it follows the rules? Pause, and decide. A: 3.42 plus or minus 0.04 seconds. The uncertainty starts with a 3, so it keeps one figure, 0.04, and the value stops at the same place, the hundredths. B is tempting, because it looks careful. But two figures are allowed only when the uncertainty starts with a 1. C reports digits the uncertainty says are meaningless.

Full Report

Checking for Systematic Error

One more line belongs in a full report: how you checked for systematic error. For the rod, the ruler was checked against a reference ruler, and no offset was found. That line matters. Statistics describe random error, but systematic error has to be hunted down, by checking equipment and methods.

Worked Example

A 0.4-Point Gap Inside the Noise

Now back to model A and model B. Is their difference larger than its own uncertainty? The difference is 0.4 points. Each result is uncertain by about 2.5 points. So by the sum rule, the difference is uncertain by the square root of 2.5 squared plus 2.5 squared: about 3.5 points. A gap of 0.4, against an uncertainty of 3.5, is well inside the noise. A gap several times larger than its uncertainty would be convincing. This one isn't. So report honestly: the uncertainty, its kind, the sample size, and your checks for systematic error. Never drop inconvenient readings, and never claim an improvement inside the noise.

Key Takeaways

Errors, Standard Error, and Honest Reports

To sum up. Random error scatters and averages out. Systematic error shifts every reading, and averaging can't touch it. The standard error, s over the square root of n, is the uncertainty of the mean. Halving it takes four times the readings. A 95 percent interval comes from a method that's right about 95 times in 100. Combine independent uncertainties by squares and a square root, and report every value with its uncertainty.

Reflection

Two Points Better After One Run

Here's something to think about. A team reports that its new model is two points better, after a single test run. What would you want to know, before you believed it?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Handling Uncertainty and Measurement in Scientific Work

What is the difference between random error and systematic error?
Random error scatters readings unpredictably above and below the true value, while systematic error pushes every reading the same way by about the same amount. Averaging many readings shrinks random error but leaves systematic error untouched. A scale that reads half a kilogram with nothing on it is systematic and needs calibration against a known reference, not more measurements.
What is the difference between standard deviation and standard error?
Standard deviation measures the spread of single readings, while standard error measures the uncertainty of their mean and equals s divided by the square root of n. Five rod readings with s of about 0.16 cm give a standard error of about 0.07 cm. Because of the square root, halving the standard error takes four times as many readings.
What does a 95 percent confidence interval actually mean?
A 95 percent confidence interval comes from a method that captures the true value in about 95 of every 100 repeats of the experiment. It is the mean plus or minus a multiplier times the standard error, with the multiplier near 2 for many readings and 2.78 for five. Any single interval might miss the truth; the 95 percent describes the method's track record.
How can you tell if one machine learning model is really more accurate than another?
Compare the gap in accuracy with the uncertainty of that gap. Test accuracy has a standard error of the square root of p(1 - p)/n, so 91.0 percent on 500 cases is uncertain by about 2.5 points at 95 percent. Combining two such uncertainties gives about 3.5 points, so a 0.4-point gap sits well inside the noise and proves nothing.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →