Standards in Psychological Measurement: Consistency, Accuracy, Fairness

UPI Study › Courses › Abnormal Psychology › Free lessons › Lesson 10.4 Free with a free account

Standards in Psychological Measurement: Consistency, Accuracy, Fairness

Abnormal Psychology · Chapter 10: Psychological Assessment: Tools and TechniquesSection 10.4 · lesson 38 of 9711 min · free with a free account

← Cognitive, Physiological, and Brain-Based AssessmentNext lesson: Generalized Anxiety Disorder: Core Features and Treatment Options →

Earn 3 college credits for Abnormal PsychologyRecommended for credit by NCCRS · Psychology 180 · self-paced · one payment, lifetime access
Enroll in Abnormal Psychology · $250 →Download the syllabus (PDF)

In this lesson

Consistency, accuracy, and the arithmetic that decides what a positive result means.

  • Define the major forms of reliability and validity
  • Read a score as a range rather than a point
  • Work a base rate through sensitivity and specificity
  • Distinguish test bias from fairness of use

Transcript

Everything said in the video, section by section (1,641 words). Read along, or search it.

Psychological Assessment: Tools and Techniques

Standards in Psychological Measurement

Here is a result that sounds alarming, and mostly is not. A screening test with ninety percent accuracy says you have a condition. In some settings that means you almost certainly do. In others it means you almost certainly do not. Same test. Same numbers. Completely different meaning. This section is about why, and about the standards that make measurement usable at all. Reliability, which is consistency. Validity, which is measuring the intended thing. Measurement error, and why a score is a band rather than a point. And the difference between a biased test and an unfair use of one. Every number you will hear is hypothetical, constructed for instruction.

Why Standards

Careful Thinking Cannot Repair Bad Measurement

Every clinical conclusion inherits the quality of the measures it rests on. A formulation built on an unreliable measure is unreliable. A diagnosis supported by an instrument that does not measure what it claims is unsupported. However carefully the reasoning was conducted afterwards. That last point is worth pausing on. Careful thinking cannot repair bad measurement. Because the error enters before the thinking starts. These standards apply to any measure, not only to formal tests. The interviews, the observations, and anything you meet later in the course. One note first. This material is educational, and is not a basis for diagnosis. And every number in it is hypothetical, constructed for instruction.

Reliability

Consistent, Which Is Not the Same as Correct

Reliability is consistency of measurement. The extent to which a measure gives the same result when nothing has changed. Notice what that is a question about. The instrument. Test-retest reliability asks whether the same person scores similarly on two occasions. Internal consistency asks whether the items are all tapping the same underlying thing. That matters whenever items get summed into a total. Sum items that measure different things, and the total measures nothing clearly. Interrater reliability asks whether two clinicians scoring the same material agree. And parallel-forms reliability matters where repeated testing risks practice effects. Now the point that ties them together. A measure is not reliable in general. It is reliable in a particular respect, and the respect has to match the use. Test-retest is essential for an ability measure, and misleading for a mood measure.

Error

A Score Is a Band, Not a Point

No measurement is exact, and that is not a failure. Measurement error is the gap between the score obtained and the score that best represents someone. Fatigue, distraction, a lucky guess, the particular items included, a room that is too warm. The standard error of measurement estimates how much a score bounces around on retesting. And it converts a score into a range. Take a hypothetical measure with a standard error of about four points. A person scoring ninety-eight is not reported as ninety-eight. They are reported as a range of roughly ninety to a hundred and six. That band is called a confidence interval, and its width comes from the reliability. Two consequences follow, and both are routinely violated. Small differences between scores are usually not differences at all. Ninety-eight and a hundred and three cannot be distinguished here. The bands overlap.

Validity

Measuring What Is Intended, for This Purpose

Validity is the extent to which a measure assesses what it is intended to assess. And the extent to which the conclusions drawn from its scores are justified. Validity is a property of an interpretation for a purpose. The same measure can be valid for one use, and invalid for another. Content validity asks whether the items cover the whole of what is being measured. Criterion validity asks whether it agrees with an established standard. Construct validity is the overarching question, answered by accumulating evidence. Then two that belong together. Convergent and discriminant. Convergent asks whether it correlates with things it should be related to. Discriminant asks whether it fails to correlate with things it should be distinct from. And discriminant is the harder and more informative test.

How They Relate

Reliable and Wrong Is a Real Possibility

Reliability is a precondition for validity, and the logic is straightforward. A measure giving different answers each time cannot be consistently measuring anything. But the reverse does not hold, and this is where people go wrong. A measure can be highly reliable and completely invalid. The clinically important case is subtler than that, though. Imagine a measure with excellent internal consistency and test-retest stability. It assesses general distress, and it is labeled as a measure of one specific condition. Every reliability statistic looks good. And every conclusion drawn about that specific condition is unsupported. Norms carry their own standards. Is the reference group the right one? Are the norms current? Population performance shifts over time.

Detection

Catching and Clearing

Two properties describe how well a test detects a condition. Sensitivity is how good it is at catching people who do have the condition. Of everyone who has it, what proportion does the test correctly identify? Specificity is how good it is at clearing people who do not have it. Of everyone without it, what proportion does the test correctly pass over? Now a worked example with entirely hypothetical figures. Suppose a screening measure has a sensitivity of ninety percent. Apply it to a thousand people, of whom a hundred have the condition. Of those hundred, the test correctly identifies ninety, and misses ten. Of the nine hundred who do not have it, it correctly passes over eight hundred and ten. Those are respectable properties, and they are properties of the test. They do not change when the test is used somewhere else.

The Base Rate

The Same Test, Two Very Different Answers

But that is not the question a person actually asks. Of everyone who tests positive, what proportion actually has the condition? The base rate. How common the condition is in the population being tested. Take the same test into a specialist clinic where forty in a hundred are affected. Out of ten thousand people, four thousand have the condition. The test finds three thousand six hundred of them, and wrongly flags six hundred others. So of four thousand two hundred positives, about eighty-six in a hundred are correct. Now take the identical test into general screening, where one in a hundred is affected. Out of ten thousand, only a hundred have the condition, and nine thousand nine hundred do not. The test finds ninety true cases. And it wrongly flags nine hundred and ninety people. So of one thousand and eighty positives, about eight in a hundred are correct. Eighty-six in a hundred, versus eight in a hundred. Same test. A positive screen in a low-prevalence setting is a reason for assessment. Not a conclusion.

The Cut-Off

Not a Statistical Question

A cut-off is the score at or above which a result is treated as positive. The score distributions of people with and without a condition overlap. Moving the line trades one kind of error for the other. Any line that reduces one error enlarges the other, as long as the distributions overlap. Which means the correct position is not a statistical question at all. It is a question about the consequences of each error, in this particular use. Where missing a case is serious and a false positive costs a further conversation, set it low. Accept many false positives to catch nearly every case. That is the logic of screening. Where a positive result triggers something burdensome, intrusive, or stigmatizing, set it higher. So the same instrument may warrant different cut-offs in different settings. And a published cut-off is a recommendation made for a purpose.

Adding Measures

More Numbers Is Not More Accuracy

Incremental validity asks whether an additional measure improves on what is already known. And it is a harder standard than it looks. Because a measure can be perfectly valid and still add nothing. So adding tests is not automatically better. Each one costs the person's time and energy, and each carries its own error. A battery assembled without this standard gives you more numbers. Response bias is a systematic tendency to answer a particular way, whatever the item says. Inconsistent responding, exaggeration, minimization, and social desirability. Validity scales are built to detect those, and here is the interpretive point. Such a finding describes the testing situation, and should prompt the question of why. Someone assessed where consequences attach has a comprehensible reason to present a certain way.

Bias And Fairness

A Technical Property, and a Judgment About Use

Two words that get used interchangeably, and mean different things here. Test bias is a technical property of a measure. Fairness is a judgment about use. A measure can be technically unbiased and used unfairly. The reverse is possible too. Predictive bias means the measure predicts an outcome differently in different groups. Item bias means individual items function differently, even where totals look similar. And construct non-equivalence means the concept itself does not mean the same thing. That is the deepest problem, because no adjustment to scoring repairs it. One error in this area has a precise form, and it is very common. Treating a translated instrument as though it were a validated one. A group difference in scores is a starting point for investigation. It is never, on its own, evidence of a group deficit.

One Composite

A Positive Screen That Is Not Information

A workplace well-being program screens all ten thousand employees. The measure has a sensitivity of ninety percent and a specificity of ninety percent. And the condition affects about one in every hundred people in this population. Rosa is thirty-six. She screens positive, and is understandably alarmed. Work the numbers. About a hundred of the ten thousand have the condition. The measure catches ninety of them, and wrongly flags nine hundred and ninety others. So one thousand and eighty people screen positive. Of whom about ninety actually have the condition. Roughly eight in a hundred. Rosa is more than eleven times more likely not to have the condition than to have it. The measure performed exactly as designed. Nothing about it failed. The program screened a population in which the condition is uncommon. Which guarantees that false positives outnumber true ones. And the result was communicated as a finding, rather than as a reason for assessment. So ask how common a condition is in the population tested, before asking about the person.

About Abnormal Psychology at UPI Study

Abnormal Psychology (Psychology 180) is a self-paced online course of 97 video lessons in 25 chapters, from the main theoretical perspectives and clinical assessment to anxiety, mood, psychotic and personality disorders, treatment and the law. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Why take it here

Why students take Abnormal Psychology with UPI Study

Understand mental health disordersAbnormal Psychology covers how psychological disorders are defined, assessed, explained and treated. It is a core course for psychology majors and useful for nursing, social work, counseling and criminal justice.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Abnormal Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Standards in Psychological Measurement: Consistency, Accuracy, Fairness

What is the difference between reliability and validity in psychological testing?
Reliability is the consistency of a measure, while validity is whether it measures what it is intended to measure and whether conclusions drawn from it are justified. Reliability is a precondition for validity, but not the reverse: a measure can be highly consistent and still measure the wrong thing, such as general distress labeled as one specific condition. Validity belongs to an interpretation for a particular purpose, not to the test itself.
What is the difference between sensitivity and specificity?
Sensitivity is the proportion of people with a condition whom a test correctly identifies, while specificity is the proportion of people without the condition whom it correctly clears. A test with 90 percent sensitivity catches 90 of every 100 true cases and misses 10. Unlike the chance that a positive result is correct, they do not depend on how common the condition is in the group tested.
Why do base rates matter when interpreting a positive screening result?
Base rates matter because the chance that a positive result is correct depends on how common the condition is in the group tested. A test with 90 percent sensitivity and specificity, used where 40 in 100 people are affected, gives correct positives about 86 percent of the time. Used where 1 in 100 are affected, false positives outnumber true ones, and only about 8 in 100 positives are correct.
What is the difference between test bias and test fairness?
Test bias is a technical property of a measure, while fairness is a judgment about how the measure is used. A test can be technically unbiased yet used unfairly. Bias can take the form of predictive bias, item bias, or construct non-equivalence, where the concept itself means something different across groups and no scoring adjustment can repair it. A group difference in scores is a starting point for investigation, never by itself evidence of a deficit.

About the Abnormal Psychology course

How many college credits is Abnormal Psychology?
Abnormal Psychology (Psychology 180) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Abnormal Psychology cost and what is included?
$250, paid once, with lifetime access: all 97 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Abnormal Psychology take?
It is self-paced with no deadlines. The 97 lessons total about 24 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Abnormal Psychology.

97 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Abnormal Psychology →Download the syllabusEvery course with PRO →
Abnormal Psychology
3 credits · NCCRSEnroll →