Personality Research Design: Validity, Reliability, and Practical Outcomes

UPI Study › Courses › Psychology of Personality › Free lessons › Lesson 2.11 Free with a free account

Personality Research Design: Validity, Reliability, and Practical Outcomes

Psychology of Personality · Chapter 2: Methods and Ethics in Personality ResearchSection 2.11 · lesson 16 of 838 min · free with a free account

← Selecting and Applying Personality Assessment TechniquesNext lesson: APA Code of Ethics: Standards and Applications in Personality Research →

Earn 3 college credits for Psychology of PersonalityRecommended for credit by NCCRS · Psychology 300 · self-paced · one payment, lifetime access
Enroll in Psychology of Personality · $250 →Download the syllabus (PDF)

In this lesson

A measure can be perfectly consistent and consistently wrong.

  • Distinguish reliability from validity
  • Describe three forms of reliability evidence
  • Explain why reliability is necessary but not sufficient
  • Apply a sound measure within its validated range

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,077 words). Read along, or search it.

Methods and Ethics in Personality Research

Validity, Reliability, and Practical Outcomes

Imagine a bathroom scale that reads three kilos heavy, every single time. It is perfectly consistent. It is also perfectly wrong. That combination has a name in measurement, and this section is about it. We will separate reliability from validity, and see how they relate. Look at three forms of reliability evidence, and three kinds of validity evidence. See why reliability is necessary but never sufficient. And why even a sound measure has to be kept inside the range it was validated for.

Consistency

Three Ways to Ask Whether a Measure Holds Still

Reliability is the consistency of a measure. A reliable instrument yields similar results across occasions, items, or raters. That gives three common forms of reliability evidence. Test-retest reliability asks whether scores are stable over time. Measure the same person twice, and see whether the score survives the gap. Internal consistency asks whether the items within a scale cohere. Do they behave as though they are measuring one thing? And inter-rater reliability asks whether independent observers agree. Looking at the same material, do they arrive at the same code? Three questions, three different threats to consistency.

The Floor

Without It, Nothing Downstream Works

Reliability is checked first, and there is a structural reason for that. Without reliability, scores fluctuate unpredictably. And they cannot support trustworthy conclusions of any kind. Consider what an unreliable measure does to a comparison. A difference between two people, or two occasions. May simply be the instrument moving, rather than the person. And you have no way to tell which happened. Every later question assumes the score is stable enough to be a score. Does it predict anything. Does it mean what we think it means. None of those questions is answerable if the number will not hold still.

Aim

Validity Asks a Harder Question

Validity asks a harder question than reliability does. It is the degree to which a measure assesses what it claims to assess. Three kinds of evidence contribute to it. Content validity asks whether the instrument covers the domain it claims to cover. Criterion validity asks whether it predicts relevant outcomes. And construct validity asks whether it fits within a coherent theoretical network. That is, does it relate to other measures the way the theory says it should? Notice something these three have in common. None of them can be settled by inspecting the instrument itself. Validity is established by what the scores do. Not by how sensible the items look on the page.

The Quadrants

One Corner of This Space Is Empty

Picture reliability on one axis, and validity on the other. In the far corner sits the usable measure. Consistent, and measuring the right thing. In the high reliability, low validity corner sits something more dangerous. A measure that is consistently wrong. And low on both, a measure that is simply fluctuating noise. Now look at the fourth corner. High validity, low reliability. It stays empty, and it has to. A measure cannot be valid without first being consistent. If the number changes at random, it cannot be tracking anything in particular.

The Relationship

Consistent Error Is Still Error

So the relationship between the two is asymmetric. Reliability is necessary, but not sufficient, for validity. A measure can be consistent, and consistently measure the wrong thing. Which is why both must be demonstrated separately. Establishing one tells you nothing at all about the other. The dangerous case is the reliable but invalid instrument. It produces stable, replicable numbers. Which is exactly what a good instrument also produces. And reliability evidence alone will never expose the problem. Because the instrument is doing the consistent thing extremely well. The error is in what it is being consistent about.

Evidence Types

What Each Check Answers

Each check answers its own question, so it is worth holding them separately. Test-retest. Is it stable over time? Indexed by correlation across occasions. Internal consistency. Do the items cohere? Indexed by an alpha coefficient. Inter-rater. Do raters agree? Indexed by an agreement statistic. Content validity. Does it cover the domain? Established by expert judgement. And criterion validity. Does it predict outcomes? Indexed by correlation with a criterion. Five rows, five separate questions, five separate answers. A strong result in one row is not evidence about any of the others. Reporting one and implying the rest is a quiet form of overclaiming.

Consequences

Weak Measurement Does Both Kinds of Damage

Measurement quality is not an internal concern of methodologists. It directly affects the practical outcomes of research. Reliable, valid measures produce findings that replicate. And findings that can be trusted to guide real decisions. Weak measurement injects error, and that error works in both directions. It can obscure real effects that are genuinely there. And it can manufacture spurious ones that are not. That second half is the part people forget. Bad measurement is not conservative. It does not merely fail to find things. So investing in measurement quality is not a technicality to be delegated. It determines whether a study yields usable knowledge at all.

Interpretation

A Sound Measure Can Still Be Misapplied

Practical outcomes also depend on appropriate interpretation. Even a sound measure must be applied within its validated range. And reported with its margin of error. There is a general principle underneath that. Validity is a property of a use, not a property of an instrument. A measure validated for one population and one purpose. Has no automatic standing outside that range. Carrying it across anyway is overreach, however good the original psychometrics were. So linking measurement quality to real-world application means three things. Acknowledging uncertainty. Avoiding overreach beyond the evidence. And ensuring decisions rest on findings the instruments can genuinely support. Which is a narrower set than the findings they can produce.

Case

The Scale That Held Together and Predicted Nothing

Here is the reliable-but-invalid case, in a real development cycle. A new resilience scale shows high internal consistency. The items cohere. The alpha coefficient looks excellent. On that evidence alone, it looks like a finished instrument. Then it fails to predict actual coping behaviour. The researchers conclude that it is reliable, but lacks criterion validity. And they revise it before using it in applied decisions. The instructive part is that the reliability evidence was never wrong. The scale was measuring something, and measuring it consistently. It just was not measuring resilience, which is what it was built to measure.

Consolidate

Demand Both, in That Order

So demand both, and in that order. Demand reliability and validity, because consistent error is still error. Establish reliability through test-retest and internal-consistency checks. Establish validity through content, criterion, and construct evidence. And keep in view what all of this is holding up. Measurement quality is the foundation everything else in a study stands on. Design, analysis and interpretation can all be excellent. And none of them can repair a measure that was measuring the wrong construct all along.

About Psychology of Personality at UPI Study

Psychology of Personality (Psychology 300) is a self-paced online course of 83 video lessons in 13 chapters, from Freud, Adler and Jung to biological, behavioral, cognitive, existential and humanistic theories, traits, personality disorders and culture. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Abnormal Psychology vs Psychology of Personality

Why take it here

Why students take Psychology of Personality with UPI Study

What makes each person who they arePsychology of Personality compares the major explanations of why people differ: psychoanalytic and neo-analytic theories, biology and evolution, learning, cognition, existential and humanistic views, and trait models, plus how personality is measured and studied. It is a common course in psychology degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Psychology of Personality carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Personality Research Design: Validity, Reliability, and Practical Outcomes

What is the difference between reliability and validity?
Reliability is the consistency of a measure, while validity is the degree to which it measures what it claims to measure. A bathroom scale that always reads three kilograms heavy is reliable but not valid: perfectly consistent and consistently wrong. Reliability is necessary but not sufficient for validity, because a measure whose scores fluctuate randomly cannot be tracking anything in particular. Both must be demonstrated separately, since good reliability alone does not show that a measure is valid.
What are the main types of reliability in psychological testing?
The three common forms are test-retest reliability, internal consistency, and inter-rater reliability. Test-retest reliability asks whether scores stay stable when the same person is measured twice, indexed by a correlation across occasions. Internal consistency asks whether a scale's items cohere, usually indexed by an alpha coefficient such as Cronbach's alpha. Inter-rater reliability asks whether independent observers looking at the same material agree, indexed by an agreement statistic.
What are content, criterion, and construct validity?
They are three kinds of evidence that a measure assesses what it claims to assess. Content validity asks whether the instrument covers the whole domain and is usually judged by experts. Criterion validity asks whether scores predict relevant outcomes, shown by correlation with a criterion. Construct validity asks whether the measure relates to other measures the way theory says it should. Validity rests on systematic evidence, such as expert review of domain coverage or how scores relate to outcomes and other measures, not on whether the items merely look relevant.
Why does weak measurement matter in personality research?
Weak measurement matters because it injects error that can both hide real effects and create spurious ones. For example, a new resilience scale might show excellent internal consistency yet fail to predict actual coping behavior, meaning it is reliable but lacks criterion validity and needs revision before applied use. Even a sound measure should be used only within the population and purpose it was validated for and reported with its margin of error.

About the Psychology of Personality course

How many college credits is Psychology of Personality?
Psychology of Personality (Psychology 300) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Psychology of Personality cost and what is included?
$250, paid once, with lifetime access: all 83 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Psychology of Personality take?
It is self-paced with no deadlines. The 83 lessons total about 8 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Psychology of Personality.

83 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Psychology of Personality →Download the syllabusEvery course with PRO →
Psychology of Personality
3 credits · NCCRSEnroll →