Reliability and Validity: The Logical Relationship, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 9.7 Free with a free account

Reliability and Validity: The Logical Relationship

Research Methods in Psychology · Chapter 9: Measurement, Reliability, and ValiditySection 9.7 · lesson 56 of 949 min · free with a free account

← Types of Reliability and How to Strengthen ThemNext lesson: Measurement Trade-Offs and Evaluating Measure Quality →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

How consistency and accuracy constrain each other, and what unreliability costs a finding.

  • Explain why reliability is necessary but not sufficient
  • Show why validity implies some reliability
  • Describe how measurement error attenuates relationships
  • Apply the disattenuation correction to a correlation

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,225 words). Read along, or search it.

Measurement, Reliability, and Validity

Reliability and Validity: The Logical Relationship

Reliability and validity are usually taught as two separate checkboxes. They are not separate. One constrains the other, in a specific and testable way. This section works out that relationship precisely. You will explain why reliability is necessary but not sufficient for validity. You will see why validity implies at least some reliability. You will learn how measurement error shrinks the relationships you observe. And you will apply a correction that estimates what the relationship really was. That correction will change how you read every modest correlation you meet.

Two Pillars

Related, but Genuinely Distinct

Reliability and validity are the two fundamental pillars of measurement quality. Reliability is the consistency, stability, and reproducibility of a measurement. Across occasions, across raters, and across items. Validity is the degree to which a measure accurately captures its intended construct. The two are related, but genuinely distinct. Reliability is a prerequisite for validity. But validity requires more than reliability alone. And this relationship has real practical consequences, not just conceptual tidiness. Unreliable measures produce attenuated correlations that underestimate true relationships. Understanding the interplay lets you diagnose measurement problems. It lets you correct for their effects, and choose sensibly among trade-offs.

Proposition One

Necessary, but Not Sufficient

The first proposition is that reliability is necessary but not sufficient. Take the necessary half first. A measure cannot be valid if it is unreliable. Inconsistent measurement simply cannot be accurate. If a scale gives different readings each time on the same person, some readings must be wrong. That is arithmetic, not opinion. Now the not sufficient half, which is where people go wrong. A measure can be highly consistent while measuring the wrong thing entirely. So high reliability does not guarantee validity. The two must be established independently, with different kinds of evidence. A reported alpha tells you nothing whatsoever about the second one.

Proposition Two

Validity Implies Some Reliability

The second proposition runs in the other direction. Validity implies at least some degree of reliability. Here is the reasoning, and it is worth following slowly. If a measure truly captures the target construct, its scores reflect genuine variation in it. And genuine variation is, by definition, consistent. So a perfectly valid measure would also be perfectly reliable. The two ideals are not independent of each other. One entails the other, in this direction at least. In practice, of course, all measures contain some error. So perfect validity and perfect reliability are unattainable ideals. The practical question is never whether you have error. It is how much you are carrying.

The Classic Illustration

Three Bathroom Scales

The bathroom scale makes all of this concrete, so take three of them. The first consistently reads five kilograms too heavy. Same wrong answer every time you step on it. That scale is perfectly reliable, and it is not valid. The second gives a different reading every time the same person steps on it. That scale is unreliable, and therefore it cannot be valid. You could never know which reading, if any, reflects true weight. The third is both consistent and accurate, and only that one is reliable and valid. Now look again at the middle case, because it proves the first proposition. Unreliability does not merely reduce validity by some amount. It removes the very possibility of establishing it.

The Practical Cost

How Unreliability Attenuates Correlations

Now the practical cost, which is the most useful idea in this section. One major consequence of measurement error is the attenuation of observed correlations. Suppose two variables are each measured with less than perfect reliability. The correlation between their observed scores will be lower than the true correlation. Lower than the real relationship between the underlying constructs. And note the word systematically. This is a bias, not noise. It does not average out across studies. It pushes in one direction, always toward zero. Which means every study using imperfect measures, and that is every study. Reports a relationship weaker than the one that actually exists.

The Formula

Attenuation, Stated Precisely

Classical Test Theory states this relationship precisely. The observed correlation equals the true correlation, multiplied by a square root term. That term is the square root of the two reliability coefficients multiplied together. Take the pieces one at a time. The observed correlation is what you actually measured between the two variables. The true correlation is the relationship between the underlying constructs themselves. And the two reliability coefficients are the reliabilities of your two measures. Cronbach's alpha, for instance, for each of them. Now read that square root term, because it carries the whole message. Reliabilities are always below one when a measure is imperfect. So the square root of their product is below one too. And multiplying by something below one always makes a number smaller.

The Correction

Disattenuation

Rearrange that equation and you get the disattenuation formula. The true correlation equals the observed correlation, divided by the same square root term. Where before you multiplied, now you divide. Disattenuation is a statistical correction. It estimates the true correlation between constructs. By removing the attenuating effect of measurement error. But be careful about what it delivers, because this is often overstated. It is an estimate, not a measurement. It tells you what the relationship would look like with perfect instruments. It does not give you perfect instruments, and your data are still what they are.

Worked Example

Mindfulness and Anxiety

Work through a real example. A researcher finds a correlation of point three zero between two scales. A mindfulness scale with an alpha of point seven five. And an anxiety scale with an alpha of point eight zero. Multiply those two reliabilities together and take the square root. That gives point seven seven five. Now divide the observed correlation by it. Point three zero divided by point seven seven five gives point three nine. So the true relationship is approximately point three nine. Measurement error in the two scales attenuated it down to point three zero. The true relationship is about thirty percent stronger than what was observed. And notice how modest the unreliability was here. Both of those alphas are conventionally acceptable. Nobody would flag either one. And they still cost about a quarter of the effect.

Consequences

Four Implications for Research

Four implications follow, and they matter for how you read published work. First, studies using low-reliability measures systematically underestimate effect sizes. And correlations along with them. Second, null results may reflect measurement error. Rather than the genuine absence of a relationship. Third, meta-analyses should apply disattenuation corrections. Particularly when pooling results across studies with different reliability levels. Otherwise the studies with worse measures drag the pooled estimate down. Fourth, improving reliability is one of the most efficient ways to increase statistical power. Often cheaper than recruiting more participants. The second implication is the one that should change your reading habits. A null finding from a study with weak measures is not evidence of no effect. It is evidence that the study could not have detected one.

Carrying It Forward

The Vocabulary of Measurement Error

Three terms to carry forward. The true score is the part of an observed score reflecting genuine variation in the construct. Uncontaminated by error. Measurement error is the part that does not reflect true variation at all. And attenuation is the systematic underestimation of a true relationship. Caused by that error in the observed scores. Here is the reading habit worth building from all of this. When you meet a modest correlation, do not immediately conclude the relationship is weak. Look for the two reliability coefficients first. Then ask what design choices would have raised them. Which is exactly where the final section of this chapter goes.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Reliability and Validity: The Logical Relationship

Can a measure be valid if it is not reliable?
No, a measure with poor reliability cannot be valid, because accuracy requires consistency. If a scale gives a different reading each time the same person is measured, some readings must be wrong, and nobody can tell which, if any, reflects the true value. The logic also runs the other way: a measure that truly captured its construct would reflect genuine variation, which is consistent. Since every real measure contains some error, the practical question is how much error it carries.
What is the correction for attenuation?
The correction for attenuation, or disattenuation, estimates the true correlation between two constructs by dividing the observed correlation by the square root of the product of the two measures' reliabilities. It reverses the Classical Test Theory relationship in which the observed correlation equals the true correlation multiplied by that same square root. The result is an estimate of the relationship with perfect measures, not a new measurement.
How much can measurement error weaken an observed correlation?
Measurement error can weaken a correlation noticeably even when reliabilities look acceptable. Suppose two scales correlate .30 and have reliabilities of .75 and .80. Their product is .60, whose square root is about .775, and .30 divided by .775 is about .39. The estimated true relationship is therefore roughly 30 percent stronger than the observed one.
Can unreliable measures hide a real relationship?
Yes, unreliable measures can hide a real relationship, because measurement error systematically shrinks observed correlations and effect sizes toward zero and lowers statistical power. A study with weak measures that finds nothing may simply have been unable to detect an effect. Improving reliability is often a cheaper way to raise power than recruiting more participants, and meta-analyses can apply disattenuation corrections when pooling studies with different reliabilities.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →