Measurement Trade-Offs and Evaluating Measure Quality

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 9.8 Free with a free account

Measurement Trade-Offs and Evaluating Measure Quality

Research Methods in Psychology · Chapter 9: Measurement, Reliability, and ValiditySection 9.8 · lesson 57 of 949 min · free with a free account

← Reliability and Validity: The Logical RelationshipNext lesson: What Internal Validity Means in Research →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

When raising reliability lowers validity, and how to appraise an instrument.

  • Explain why reliability and validity can pull apart
  • Evaluate four trade-offs in measure design
  • Apply an eight-question appraisal framework
  • Synthesize the chapter's account of measurement

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,185 words). Read along, or search it.

Measurement, Reliability, and Validity

Measurement Trade-Offs and Evaluating Measure Quality

Here is a counter-intuitive fact to end the chapter on. You can make a measure more reliable and, in doing so, make it worse. This section explains how that happens, and how to spot it. You will see why reliability and validity can pull against each other. You will work through four specific trade-offs in measure design. You will learn an eight-question framework for appraising any instrument. And you will pull the whole chapter together into a single account of measurement. Which turns out to be less technical, and more philosophical, than it first appears.

The Tension

Why Maximizing One Can Damage the Other

Recall the two propositions from the previous section. Reliability is necessary but not sufficient for validity. And validity implies at least some degree of reliability. From those two, it is tempting to conclude that more reliability is simply always better. In practice, it is not. Attempts to maximize reliability can sometimes reduce validity. And the reverse holds as well. So researchers have to navigate several trade-offs, deliberately rather than accidentally. The reason is structural, and once you see it the four cases become obvious. Most techniques that raise reliability work by narrowing something. The item pool, the setting, or the definition itself. And narrowing is precisely what threatens your coverage of the construct.

Trade-Off One

Breadth Versus Precision

The first trade-off is breadth against precision. Take a broad construct like wellbeing. Measuring it properly requires diverse items. And diverse items reduce internal consistency. Not because your scale is bad, but because wellbeing is genuinely multidimensional. Now watch the trap open up. You could remove the diverse items to increase alpha. Your reliability coefficient would improve immediately. But that may reduce content validity, by narrowing the construct inappropriately. Your alpha goes up, and your measure gets worse. So a low alpha on a multidimensional construct is not automatically a defect. It may be an accurate reflection of the construct's own structure.

Trade-Off Two

Standardization Versus Ecological Validity

The second trade-off is standardization against ecological validity. Highly standardized laboratory measurement maximizes reliability. It does so by controlling extraneous variation. Everyone gets the same instructions, the same room, and the same task. That control genuinely works. The scores become much more consistent. But consider what the control removes. Standardization may strip out the contextual factors that give the construct real-world meaning. You end up measuring the thing very cleanly. In a setting where the thing does not naturally occur. You have met this shape before, in the chapters on design and on sampling. Control buys precision, and spends relevance.

Trade-Offs Three and Four

Two Ways to Narrow a Construct

The third and fourth trade-offs are two ways of narrowing the same thing. The third is item homogeneity against construct coverage. Select only items that are highly intercorrelated, and internal consistency rises. But you produce a narrow, homogeneous measure. One that underrepresents the full domain of the construct. The fourth is specificity against generalizability. A highly specific operational definition may be reliably measurable. And still invalid for capturing the full, multidimensional construct. Aggression makes this concrete. Measure aggression only as physical acts, and you will measure it very consistently. You will also miss its verbal and relational forms. Which are part of the construct you claimed to be studying.

All Four

The Trade-Offs on One Page

Put the four trade-offs on one page. Breadth against precision. Diverse items lower alpha, and removing them narrows the construct. Standardization against ecological validity. Control raises reliability, and can remove the real-world meaning. Item homogeneity against construct coverage. Intercorrelated items raise alpha, and underrepresent the domain. And specificity against generalizability. A tight operational definition measures reliably, and may not be the construct at all. Look at the shape all four share. In every case, the fix for reliability is narrowing. And the cost is coverage. Recognizing that pattern is worth more than memorizing the four individual cases.

Appraisal, Part One

What Is It, and What Does It Cover?

Now a framework for evaluating any measurement instrument. Use it when selecting a measure, or when critically appraising a published paper. Eight questions, and the first four establish what the measure is. One. What construct is measured, and how is it theoretically defined? If nobody can answer that, nothing after it will help. Two. What content validity evidence exists? Were domain coverage and item relevance evaluated by experts? Three. What internal consistency evidence is reported? Is the reported alpha adequate for the intended use of the scores? Four. What test-retest reliability evidence exists? And does the stability shown match the construct's expected stability?

Appraisal, Part Two

Does It Behave as It Should?

The second four questions ask whether the measure behaves as it should. Five. What criterion validity evidence exists? Does the measure relate to the external standards you would expect? Six. What construct validity evidence exists? Convergent and discriminant correlations, known-groups differences, factor structure. Seven. Has the measure been validated in the specific population under study? By age, by culture, and by language. Eight. Are there known sources of bias that might threaten validity in this context? Of those eight, question seven is the one most often skipped. Psychometric evidence travels less well than people assume. A measure validated somewhere else is not automatically validated here.

The Chapter's Arc

Where This Chapter Has Taken You

Step back and look at the arc this chapter has traced. It began with foundations. The distinction between qualitative and quantitative approaches to measurement. Then translation. Turning abstract constructs into operational definitions. Definitions precise enough that another researcher could replicate them exactly. Then levels. Nominal, ordinal, interval, and ratio. And the statistics each level does and does not permit. Then instruments. The major categories and forms of measurement tool. And finally the twin pillars, reliability and validity, and the interplay between them. Each stage constrains the ones after it. Which is why a mistake early in that chain cannot be repaired later.

The Chapter's Arc

Eight Things to Take Away

Eight things worth taking away from the whole chapter. Measurement is the systematic assignment of numbers or categories by defined rules. Qualitative and quantitative approaches capture different aspects of a phenomenon. They suit different research questions, and neither is superior. Operational definitions translate abstract constructs into measurable procedures. Precision and construct validity are essential to them. The four levels of measurement determine which statistical analyses are appropriate. Self-report, behavioral, physiological, implicit, and archival forms all have distinct strengths. Reliability is the consistency of measurement, necessary but not sufficient for validity. Validity is accuracy, the degree to which a measure truly captures its intended construct. And unreliability attenuates correlations, while disattenuation corrects for that effect.

The Central Message

Measurement Is Not a Neutral Procedure

The chapter ends on a claim that is larger than any of its techniques. Measurement is not a neutral technical procedure. Every measurement decision embeds assumptions. What construct to assess. How to define it. Which instrument to use. How to administer it, and how to score it. Each of those choices carries assumptions about the nature of mind. And about the nature of knowledge itself. So those assumptions have to be made explicit, and examined critically. And they have to be supported by evidence. Only then can the data from psychological research be trusted. Trusted to reveal something true about the phenomena we seek to understand. One line to carry out of the chapter, if you carry only one. A study can be no better than the instrument that produced its numbers. And that instrument is a set of choices, not a fact of nature.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Measurement Trade-Offs and Evaluating Measure Quality

Can making a measure more reliable make it less valid?
Yes, steps that raise reliability can lower validity, because they usually work by narrowing something: the item pool, the setting, or the definition of the construct. Narrowing threatens coverage of the construct. Measuring aggression only as physical acts, for example, may give very consistent scores while missing verbal and relational aggression, which are part of the construct being studied.
How does standardization affect ecological validity?
Highly standardized laboratory measurement raises reliability by giving everyone the same instructions, room, and task, but it can lower ecological validity. The same control that makes scores consistent may strip away the contextual factors that give a construct its real-world meaning. The researcher ends up measuring the behavior very cleanly in a setting where it does not naturally occur, so control buys precision at the cost of relevance.
Is a low Cronbach's alpha always a bad sign?
No, a low Cronbach's alpha is not automatically a defect when the construct is genuinely multidimensional. A broad construct such as wellbeing needs diverse items, and diverse items lower internal consistency. Deleting them to push alpha up can damage content validity by narrowing the construct. In that case a modest alpha may accurately reflect the construct's structure rather than a poorly built scale.
How do you evaluate the quality of a psychological measure?
A thorough evaluation asks eight questions: what construct is measured and how it is defined; what content validity evidence exists; whether internal consistency is adequate for the intended use; whether test-retest stability matches the construct's expected stability; what criterion validity evidence exists; what construct validity evidence exists, such as convergent and discriminant correlations and factor structure; whether the measure was validated in the population studied; and what sources of bias could apply.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →