Reliability and the Sources of Measurement Error, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 9.5 Free with a free account

Reliability and the Sources of Measurement Error

Research Methods in Psychology · Chapter 9: Measurement, Reliability, and ValiditySection 9.5 · lesson 54 of 949 min · free with a free account

← Exploring the Different Categories of Research TestsNext lesson: Types of Reliability and How to Strengthen Them →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

What consistency means, what destroys it, and why it limits every conclusion.

  • Define reliability and explain why it matters
  • Explain attenuation due to measurement error
  • Apply the Classical Test Theory decomposition
  • Identify the five major sources of measurement error

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,215 words). Read along, or search it.

Measurement, Reliability, and Validity

Reliability and the Sources of Measurement Error

A bathroom scale that reads five pounds heavy every time is perfectly reliable. It is also completely wrong, and those are two separate facts. This section is about the first one, consistency. You will define reliability, and see why an unreliable measure limits everything downstream. You will learn what attenuation is, and why it biases findings toward zero. You will apply the Classical Test Theory decomposition of an observed score. And you will identify the five places measurement error actually comes from. That last one is the practical payoff, because error is fixable and reliability is not, directly.

The Definition

Consistency, Stability, Reproducibility

Reliability refers to the consistency, stability, and reproducibility of a measurement. Three words for one idea. A reliable measure produces the same result under the same conditions. Measure the same phenomenon twice, under identical circumstances. A reliable instrument gives you the same result, or a very similar one, both times. Now notice what that definition does not claim. It says nothing about whether the measure is correct. A consistently wrong instrument is perfectly reliable. Accuracy is an entirely separate question, and it has its own name, validity. Keep the two apart, because almost every confusion in this area comes from merging them.

The Cost

What Unreliability Does to a Study

So what does unreliable measurement actually cost you? It introduces noise into your research data. It reduces the power to detect real effects. And it ultimately limits what can be concluded from the study at all. Pay attention to the direction of that damage. Unreliability does not usually manufacture effects that are not there. It buries effects that are, under a layer of random variation. Which is precisely what makes it so hard to notice. Your study returns a null result, and a null result looks like a finding. Nothing in the output announces that the instrument was the problem.

The Deepest Consequence

Attenuation

There is a deeper consequence, and it links reliability to validity directly. An unreliable measure cannot be a valid measure. That is not a slogan. It follows from what unreliability does to a correlation. If a scale does not produce consistent scores, watch what happens. Any apparent relationship between its scores and other variables gets attenuated. Which means systematically underestimating the true relationship. The phenomenon is called attenuation due to measurement error. And it is a fundamental problem in psychological research. Note the word systematically, because it changes the nature of the problem. Your effect estimate is not merely uncertain. It is biased toward zero. And that bias grows steadily as reliability falls.

A Common Misconception

Reliability Belongs to Scores, Not to Tests

Now a misconception that costs researchers a great deal. Reliability is a property of scores in a specific context. It is not a property of a test itself, in the abstract. People speak as though reliability were printed on the instrument, like a serial number. It is not. A scale that is highly reliable in one population or setting may be much less reliable in another. Different age group, different language, different level of the construct. So reliability evidence must be gathered for each new application of a measure. Saying the manual reports an alpha of point eight nine tells you about somebody else's sample. It does not tell you about yours.

The Framework

Classical Test Theory

The framework underneath all of this is Classical Test Theory. It conceptualizes every observed score as having exactly two components. The true score, which is the actual level of the construct in that participant. And measurement error, which is everything else that moved the score. From that decomposition, one relationship follows immediately. Reliability is inversely related to measurement error. The more error, the lower the reliability. There is no third term. Which means improving reliability always means removing a source of error. And that reframing is genuinely useful, because error is concrete. You cannot chase reliability directly. Nobody knows what that would look like. But you can identify a noisy room, an ambiguous item, or an untrained coder.

Source One and Five

Error That Comes from the Participant

So where does the error come from? Five places, and two of them are the participant. The first is temporary participant characteristics. Mood, fatigue, hunger, motivation, and physical health all fluctuate over time. And they can affect responses in ways unrelated to the construct being measured. A tired participant scores differently from a rested one, on the same underlying trait. The fifth source is chance responding. Participants make random errors. Misreading an item, pressing the wrong key. Simply losing concentration for a moment. Those produce scores unrelated to the participant's true construct level. Notice that these two behave quite differently, and that matters later. Temporary characteristics shift a whole session in one direction. Chance responding scatters unpredictably across individual items.

Source Two

Error Built into the Items

The second source is built into the instrument itself. Item characteristics. Poorly worded, ambiguous, or double-barreled items introduce error. They do it because different participants interpret them differently. The double-barreled item is the classic trap, so here is one. I find my work interesting and rewarding. That asks two questions, and accepts only one answer. Someone who finds the work interesting but unrewarding has no honest response available. Whatever they choose adds error to your data. This source deserves attention for a specific reason. It is the one entirely under the researcher's control. And it is the cheapest to fix, provided you fix it before data collection starts.

Sources Three and Four

Error from the Setting and the Researcher

The third and fourth sources come from outside the participant entirely. Testing conditions are the third. Variations in the environment, such as noise, lighting, temperature, and time of day. These can affect scores either systematically or randomly. The fourth source is interviewer and observer effects. In interview or observational studies, different people run the session differently. They may ask questions differently, or code the same behavior differently. There is something uncomfortable about that fourth source. The error is introduced by the person collecting the data, not by the participant. It is completely invisible in the dataset afterwards. And the only way to detect it afterwards is to have used more than one rater.

All Five Together

Where the Noise Comes From

Put all five sources side by side. Temporary participant characteristics, which fluctuate within a person over time. Item characteristics, where the wording itself invites inconsistent interpretation. Testing conditions, where the room and the hour move the numbers. Interviewer and observer effects, where the researcher becomes part of the instrument. And chance responding, which is simply noise from the participant's own slips. Read that list as a diagnostic checklist rather than as a taxonomy. When a measure performs poorly, one of those five is usually responsible. And each one is addressed by a different intervention. Which is exactly why there is no single fix for low reliability.

Where This Leads

Each Source Needs Its Own Check

That fact has an important consequence for how reliability gets assessed. Because error arrives from five different directions, no single number certifies a measure. Different types of reliability are sensitive to different sources of error. A check that catches instability over time will not catch disagreement between raters. Two terms to carry forward. Attenuation is the underestimation of a true relationship, caused by measurement error. In one measure or in both. Classical Test Theory is the framework treating an observed score as true score plus error. And the encouraging news is that reliability is not fixed. It can be improved through careful attention to design, administration, and scoring. Next we take the four types of reliability in turn, and the strategies that raise each one.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Reliability and the Sources of Measurement Error

What is reliability in psychological measurement?
Reliability is the consistency, stability, and reproducibility of a measurement: a reliable instrument gives the same or very similar results when the same thing is measured under the same conditions. Reliability says nothing about accuracy. A bathroom scale that always reads five pounds too heavy is perfectly reliable but wrong, and accuracy is a separate property called validity.
What does Classical Test Theory say about test scores?
Classical Test Theory treats every observed score as the sum of two parts: a true score, reflecting the person's actual level of the construct, and measurement error, reflecting everything else that moved the score. The more error a measure contains, the lower its reliability. Improving reliability therefore means finding and removing specific sources of error, such as an ambiguous item, a noisy room, or an untrained coder.
What are the main sources of measurement error?
Five common sources of measurement error are temporary participant states such as mood, fatigue, or hunger; poorly worded or double-barreled items; testing conditions such as noise, lighting, temperature, and time of day; interviewer and observer differences; and chance responding, such as misreading an item or pressing the wrong key. A double-barreled item like "I find my work interesting and rewarding" asks two questions but allows only one answer.
Is reliability a property of a test or of its scores?
Reliability is a property of scores obtained in a specific sample and setting, not a fixed property of the test itself. A scale that is highly reliable in one population can be much less reliable in a different age group, language, or range of the construct. That is why researchers should estimate and report reliability in their own sample rather than relying only on the value printed in a test manual.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →