Types of Reliability and How to Strengthen Them, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 9.6 Free with a free account

Types of Reliability and How to Strengthen Them

Research Methods in Psychology · Chapter 9: Measurement, Reliability, and ValiditySection 9.6 · lesson 55 of 9410 min · free with a free account

← Reliability and the Sources of Measurement ErrorNext lesson: Reliability and Validity: The Logical Relationship →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

The four coefficients, what each one checks, and how to raise them.

  • Distinguish the four major types of reliability
  • Interpret each coefficient against its benchmark
  • Avoid the standard misreading of Cronbach's alpha
  • Apply six strategies for strengthening a measure

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,341 words). Read along, or search it.

Measurement, Reliability, and Validity

Types of Reliability and How to Strengthen Them

A measure can be consistent in four different ways. Across time, across items, across raters, and across versions of the test. Each has its own coefficient, and its own benchmark for adequacy. This section takes all four in turn. You will interpret each coefficient against the number that counts as acceptable. You will avoid the single most common misreading of Cronbach's alpha. And you will learn six concrete strategies for raising a measure's reliability. Because reliability is not fixed. It responds to how you design and run the study.

Four Questions

Consistency Across What?

Start with a framing that makes the four types easy to keep straight. Reliability always means consistency. The question is, consistency across what? Test-retest reliability asks about consistency across time. Internal consistency asks about consistency across the items within one scale. Inter-rater reliability asks about consistency across the people doing the rating. And parallel forms reliability asks about consistency across two versions of the test. Four dimensions, four different sources of error being checked. Which is why a single reliability number never certifies a measure by itself. A study with one rater simply has no evidence about the third dimension.

Across Time

Test-Retest Reliability

Test-retest reliability assesses the stability of scores over time. The procedure is straightforward. You administer the same measure to the same participants on two occasions. Separated by an appropriate interval, which we will come back to. Then you calculate the correlation between the two sets of scores. A high correlation, typically point seven zero or above, indicates that scores are stable. Now here is a subtlety students often miss. A low test-retest correlation is not automatically bad news about the instrument. If the construct itself changed between the two occasions, low correlation is correct behavior. The interpretation depends entirely on what you were trying to measure.

Across Time

Choosing the Interval

Which brings us to the interval between administrations. The appropriate interval depends on the construct, and picking it badly produces a meaningless number. Take a stable trait, such as personality. That should show high test-retest reliability over weeks or even months. Now take a fluctuating state, such as current mood. That should show lower stability over the very same period. And that lower number is the expected result, not a defect in the scale. A mood measure with a correlation of point nine five over three months would be suspicious. But one case is genuinely diagnostic, and it is worth memorizing. If test-retest reliability is low for a measure of a stable trait, that is evidence of measurement error. It is not evidence of genuine instability in the construct.

Across Items

Internal Consistency and Cronbach's Alpha

Internal consistency reliability asks a different question entirely. To what degree are the items within a scale all measuring the same construct? Do they hang together, in the usual phrase? It is typically assessed using Cronbach's alpha. Alpha reflects the average correlation between items. Corrected for the number of items in the scale. That correction is worth pausing on, because it has a consequence. Since alpha is corrected for scale length, length also inflates it. A long scale of only weakly related items can still reach a respectable alpha. So read a reported alpha alongside the number of items it came from.

Across Items

Reading an Alpha

There are conventional benchmarks for interpreting an alpha. Below point six zero is unacceptable. Point six zero to point six nine is questionable. Point seven zero to point seven nine is acceptable. Point eight zero to point eight nine is good. And point nine zero or above is excellent. But now the part that surprises people. Higher is not simply better. Very high alpha values, above point nine five, may indicate item redundancy. The scale contains items so similar that they add little new information. In effect, you asked the same question five different ways. That is consistent, certainly. It is also wasteful of your participants' patience.

A Standard Error

Alpha Is Not a Test of Unidimensionality

Now the most common misunderstanding about alpha, and it is worth being blunt. Cronbach's alpha is often taken as a measure of unidimensionality. That is, the extent to which items measure a single underlying factor. It is not that. A high alpha can be achieved by a multidimensional scale. It only requires that the items be sufficiently intercorrelated with each other. So a good alpha is not evidence that you measured one thing. It is evidence that whatever you measured, you measured it consistently. Unidimensionality is best evaluated through factor analysis. Not through alpha alone. Two similar-sounding questions, two different tools.

Across Raters

Inter-Rater Reliability

The third type is inter-rater reliability, sometimes called inter-observer reliability. It assesses the degree to which two or more raters produce consistent scores. Working independently, on the same behavior or phenomenon. It is essential in any study using observational coding. Also in clinical rating scales, and in qualitative coding of interview data. Two coefficients are used, and which one depends on your data. For categorical ratings, use Cohen's kappa. Kappa values above point six zero are generally considered adequate. Above point eight zero is considered strong. For continuous ratings, use the intraclass correlation coefficient instead. Values above point seven five are considered good to excellent.

Across Versions

Parallel Forms Reliability

The fourth type is parallel forms reliability, also called alternate forms. It assesses the equivalence of two versions of a measure. Both designed to assess the same construct. Why would anyone build two versions of the same test? Usually to reduce practice effects in repeated testing. If you test someone twice, familiarity with the items contaminates the second score. The correlation between scores on the two forms shows how interchangeable they are. And notice that the threshold here is higher than for test-retest. A correlation of point eight zero or above is required before forms can be used interchangeably. That makes sense, because you are making a stronger claim. Not merely that scores are stable, but that the two versions are genuine substitutes.

All Four

The Coefficients at a Glance

Put all four coefficients on one page. Test-retest assesses stability over time. It uses a Pearson correlation between two administrations, and wants point seven zero or above. Internal consistency assesses the consistency of items within a scale. It uses Cronbach's alpha, and again wants point seven zero or above. Inter-rater assesses agreement between observers or raters. It uses Cohen's kappa or the intraclass correlation coefficient. Kappa wants point six zero or above, and the intraclass coefficient wants point seven five. Parallel forms assesses the equivalence of two test forms. It uses a Pearson correlation between the forms, and wants point eight zero or above. Treat those benchmarks as conventions rather than laws, but know them. Reviewers certainly do.

Raising It

Six Strategies for Strengthening Reliability

Reliability is not fixed. It can be improved, and six strategies do most of the work. First, increase the number of items. Random errors in individual items cancel out across a larger item pool. The Spearman-Brown prophecy formula estimates how many items you need for a target reliability. Second, improve item clarity. Ambiguous, double-barreled, or leading items introduce error directly. Pilot testing with cognitive interviews checks that items are understood as intended. Third, standardize administration. Identical scripted instructions and testing conditions reduce error from situational variability. Fourth, train and calibrate your observers. Rigorous training, detailed coding manuals, and regular calibration sessions. Those maintain inter-rater reliability over the life of a project. Fifth, use validated instruments where they exist. And sixth, assess and report reliability in every study you run. Do not assume that reliability established in a previous sample applies to your own.

The Limit

Necessary, but Not Sufficient

Two closing cautions. The first is about that first strategy, adding items. Reliability increases with the number of items, but only within limits. Adding near-duplicate items eventually buys you redundancy rather than precision. The second caution is the sentence this whole chapter turns on. Reliability is essential, but it is not sufficient for validity. A measure can be perfectly consistent without being accurate about anything at all. Think again about the scale that reads five pounds heavy every single time. So a strong alpha earns you the right to ask the next question, and no more. That question is whether the instrument measures the construct it claims to measure. And that is exactly where the chapter goes next.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Types of Reliability and How to Strengthen Them

What are the four main types of reliability?
The four main types of reliability each check consistency across something different. Test-retest reliability checks consistency over time, usually wanting a correlation of .70 or higher. Internal consistency checks agreement among the items of a scale, usually with Cronbach's alpha. Inter-rater reliability checks agreement between independent raters. Parallel-forms reliability checks whether two versions of a test are interchangeable, typically requiring a correlation of .80 or higher.
What is a good Cronbach's alpha value?
A Cronbach's alpha of .70 or higher is conventionally considered acceptable, .80 to .89 good, and .90 or above excellent, while values in the .60s are often called questionable. Higher is not always better: values above about .95 may signal redundant items that ask the same question in slightly different words. Alpha also rises with the number of items, so read it alongside the length of the scale.
Does a high Cronbach's alpha mean a scale is unidimensional?
No, a high Cronbach's alpha does not show that a scale measures a single construct. Alpha depends on how strongly the items intercorrelate and on how many items there are, so a multidimensional scale can still reach a high value. A good alpha shows that the items were answered consistently. Whether they reflect one underlying factor is better tested with factor analysis.
When should you use Cohen's kappa instead of the intraclass correlation coefficient?
Use Cohen's kappa for inter-rater reliability when raters assign categorical codes, and the intraclass correlation coefficient (ICC) when ratings are continuous. Kappa values above .60 are generally considered adequate and above .80 strong, while ICC values above .75 are considered good to excellent. Rater agreement can be kept high through thorough training, detailed coding manuals, and regular calibration sessions.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →