What Makes a Good Test, Actually? Qualities, Validity, and Reliability

UPI Study › Courses › Educational Psychology › Free lessons › Lesson 8.3 Free with a free account

What Makes a Good Test, Actually? Qualities, Validity, and Reliability

Educational Psychology · Chapter 8: Assessment, Testing, and Making Sense of ResultsSection 8.3 · lesson 72 of 1009 min · free with a free account

← Checking In vs. Wrapping Up: Formative and Summative AssessmentNext lesson: Did They Learn It or Just Do It? Product vs. Process Assessment →

Earn 3 college credits for Educational PsychologyRecommended for credit by NCCRS · Psychology 120 · self-paced · one payment, lifetime access
Enroll in Educational Psychology · $250 →Download the syllabus (PDF)

In this lesson

Validity, reliability, and why one of them cannot exist without the other.

  • Define validity and name three types of evidence for it
  • Define reliability and name three forms of it
  • Explain why reliability is necessary but not sufficient
  • Improve inter-rater reliability on a performance task

Transcript

Everything said in the video, section by section (1,176 words). Read along, or search it.

Assessment, Testing, and Making Sense of Results

What Makes a Good Test, Actually?

Two words decide whether a test result means anything at all. Validity, and reliability. They sound like synonyms, and they are not. One of them cannot exist without the other, but not the other way around. This section works through both, and the relationship between them. We will look at three kinds of validity evidence. Then three forms of reliability, and how each is measured. Then the asymmetry that catches people out. And we will end with a teacher fixing a rubric in one afternoon. Her agreement with a colleague went from sixty percent to eighty-eight. Nothing about the essays changed. Only the scoring did.

Validity

Does It Measure What You Think It Measures?

Validity refers to the degree to which a test measures what it intends to. That definition sounds obvious until you see it fail. Consider a mathematics exam built from dense word problems. Paragraph-length word problems, all the way through. That exam may actually be measuring reading comprehension. More than it measures mathematical reasoning. Which undermines its validity as a math assessment. Now notice what is not wrong in that example. The questions themselves may be perfectly good questions. The problem is a mismatch between what is measured and what is claimed. A low score will be reported as weak mathematics. And for some students, that report will simply be false.

Not Binary

Validity Is a Continuum, and It Has an Address

Validity is not a binary property, and this matters more than it sounds. A test is not simply valid, or invalid. Validity exists on a continuum. And it must be established for a specific purpose and population. The authoritative source here is the Standards for Educational and Psychological Testing. It is published jointly by three organizations. The American Educational Research Association. The American Psychological Association. And the National Council on Measurement in Education. The Standards describes validity as a unitary concept. A concept supported by multiple types of evidence. So the right question is never simply, is this test valid? It is: valid for what, and valid for whom?

Three Evidences

Content, Construct, Criterion

Three classic types of validity evidence are worth knowing. Content validity asks about coverage. Do the test items adequately sample the full domain being assessed? Suppose a chemistry exam covers only stoichiometry. But the course also addressed thermodynamics and acid-base reactions. That test lacks content validity, however well written it is. Construct validity asks a deeper question. Does the test measure the theoretical construct it claims to measure? Does an anxiety scale measure anxiety, or general negative affect? Criterion validity asks about prediction. Do scores correlate with a meaningful external criterion? For instance, do college entrance exam scores predict first-year GPA?

Reliability

Would You Get the Same Answer Twice?

Reliability is a different property entirely. It refers to the consistency of a test's scores. Consistency across repeated administrations, raters, or items. A reliable test gives similar scores for the same student. Under similar conditions, on separate occasions. Without reliability, something specific goes wrong with the numbers. Scores become dominated by measurement error. Rather than by true differences in student knowledge. That makes them useless for instructional decision-making. Measurement error is the gap between an observed score and a true score. It is not a mistake anybody made. It is noise, produced by the instrument itself.

Three Forms

Across Time, Across Raters, Across Items

There are three primary forms of reliability. Each one asks where the inconsistency could be coming from. Test-retest reliability is about time. Same test, same students, different occasions. Are scores stable? Inter-rater reliability is about people. Same responses, scored by different raters. Do they agree? Internal consistency is about the test's own items. Do the items within a test measure the same underlying construct? Which form you need depends on the threat you actually face. A writing assessment scored by humans is threatened by the second. Checking the first would tell you almost nothing useful.

Measuring It

Cronbach's Alpha, and Calibrating Humans

Internal consistency is often estimated with Cronbach's alpha. It is a coefficient that ranges from zero to one. Values above zero point eight zero are generally considered acceptable. Acceptable, that is, for educational assessments. Inter-rater reliability is especially critical elsewhere. Specifically for performance assessments and writing evaluations. Those are the places where human judgment plays a central role. Three strategies improve agreement between raters. Rubric training. Anchor papers. And calibration sessions. But there is a hard limit worth stating plainly. When raters disagree substantially, the scores are unreliable. And that holds regardless of how well the rubric was designed.

The Six

Each Question, and How to Answer It

It helps to see all six categories with their questions attached. Content validity asks whether items cover the full domain. You establish it with a test blueprint, and expert review. Construct validity asks whether the intended construct is measured. You establish it with factor analysis and convergent evidence. Criterion validity asks whether the test predicts an external criterion. You establish it by correlation with GPA, or with job performance. Test-retest asks whether scores are stable over time. You establish it by giving the same test twice, on different occasions. Inter-rater asks whether raters agree, measured with a kappa coefficient. And internal consistency asks whether items measure the same construct. There, Cronbach's alpha above zero point eight zero is the usual benchmark.

The Relationship

One Quadrant Cannot Be Occupied

Now picture reliability and validity as two axes on a grid. Reliability along the bottom, validity up the side. Three of the four corners describe real tests. High on both is the goal. That is a good assessment. High reliability with low validity is consistently wrong. It gives you the same answer every time, to the wrong question. Low on both is simply noise, and it is at least obvious. But look at the fourth corner, and think about it carefully. High validity with low reliability. That combination cannot exist, and the reason is worth understanding. A score that changes randomly cannot be accurately measuring anything. The empty corner is the whole lesson of this slide.

Necessary, Not Sufficient

The Bathroom Scale Argument

The classic illustration of the asymmetry is a bathroom scale. A bathroom scale is consistent, and therefore reliable. Step on it three times and you get the same number. Now use that scale to measure intelligence. The scores are perfectly consistent, and completely meaningless. So a test can be reliable without being valid. The reverse, however, is not true. A test cannot be valid if it is unreliable. Inconsistent scores cannot provide an accurate picture of learning. Which gives us the phrase to remember. Reliability is necessary, but not sufficient, for validity. The design goal is a test that consistently measures exactly the right thing.

In Practice

From 60 Percent Agreement to 88

Here is that theory turning into an afternoon of practical work. A high school teacher develops an essay rubric for a persuasion unit. She scores ten sample essays herself. Then she asks a colleague to score the same essays independently. When they compare, inter-rater agreement is only sixty percent. That is a substantial amount of disagreement on the same papers. She does not abandon the rubric, which is the important choice. They hold a calibration session, and discuss the borderline papers. Then they revise the criteria for clarity. On the next round, agreement rises to eighty-eight percent. Consider why that afternoon was worth spending. Grade retention, special education placement, scholarship awards. Poorly constructed assessments produce noise rather than signal.

About Educational Psychology at UPI Study

Educational Psychology (Psychology 120) is a self-paced online course of 100 video lessons in 11 chapters, from memory, conditioning and child development to motivation, individual differences, assessment, instruction and educational technology. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Why take it here

Why students take Educational Psychology with UPI Study

How people learn, and how to teach themEducational Psychology explains how students learn, develop and stay motivated, and how teachers assess and support them. It is a standard course in education and teaching degrees and a common psychology elective.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Educational Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About What Makes a Good Test, Actually? Qualities, Validity, and Reliability

What is the difference between validity and reliability?
Validity is the degree to which a test measures what it intends to measure, while reliability is the consistency of its scores across occasions, raters or items. A test can be reliable without being valid, like a bathroom scale used to measure intelligence, but it cannot be valid if it is unreliable. Reliability is necessary but not sufficient for validity.
What are content, construct and criterion validity?
Content validity asks whether test items adequately sample the whole domain, such as a chemistry exam covering every major topic taught. Construct validity asks whether the test measures the theoretical construct it claims, such as anxiety rather than general negative affect. Criterion validity asks whether scores correlate with an external criterion, such as college entrance exam scores predicting first-year GPA.
What are the three main types of reliability?
The three main types are test-retest reliability, inter-rater reliability and internal consistency. Test-retest checks whether scores are stable when the same students take the same test on different occasions. Inter-rater checks whether different raters agree when scoring the same responses, often measured with a kappa coefficient. Internal consistency checks whether a test's items measure the same construct, usually estimated with Cronbach's alpha, where values above 0.80 are generally considered acceptable.
How can teachers improve inter-rater reliability on essays or projects?
Teachers can improve agreement between raters through rubric training, anchor papers that illustrate each score level, and calibration sessions where raters discuss borderline work and clarify criteria. In one example, two teachers scoring the same ten essays agreed only 60 percent of the time; after a calibration session and clearer criteria, agreement rose to 88 percent. Substantial disagreement makes scores unreliable however good the rubric.

About the Educational Psychology course

How many college credits is Educational Psychology?
Educational Psychology (Psychology 120) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Educational Psychology cost and what is included?
$250, paid once, with lifetime access: all 100 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Educational Psychology take?
It is self-paced with no deadlines. The 100 lessons total about 15 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Educational Psychology.

100 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Educational Psychology →Download the syllabusEvery course with PRO →
Educational Psychology
3 credits · NCCRSEnroll →