Internal Threats: History, Instrumentation, Mortality, Regression, and Testing

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 10.5 Free with a free account

Internal Threats: History, Instrumentation, Mortality, Regression, and Testing

Research Methods in Psychology · Chapter 10: Internal Validity and Its ThreatsSection 10.5 · lesson 62 of 9411 min · free with a free account

← Participant-Related Factors That Affect Internal ValidityNext lesson: Internal Threats: Selection, Maturation, and Their Combined Effects →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

History, instrumentation, mortality, regression, and testing.

  • Identify the designs most vulnerable to history
  • Distinguish mechanical from human instrumentation drift
  • Explain why differential attrition is the real danger
  • Separate testing effects from genuine learning gains

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,442 words). Read along, or search it.

Internal Validity and Its Threats

Five Classical Threats to Internal Validity

Select the lowest-scoring students for a tutoring program. Test them again a year later, and their scores will be higher. That would happen even if the tutoring did absolutely nothing. This section names five threats with that character. Each produces a pattern that looks exactly like a treatment effect. History, instrumentation, mortality, regression, and testing. You will learn which designs are most vulnerable to each one. And the design feature that neutralizes it, which is usually simpler than you expect.

The Catalog

Campbell and Stanley, 1963

In nineteen sixty-three, two methodologists published a short monograph. Donald Campbell and Julian Stanley. It has shaped research methodology ever since. Their key contribution was a systematic catalog. A list of the specific threats to internal validity that recur across studies. Along with the design features that neutralize each one. The catalog has been refined over the decades. But its core remains the vocabulary in which methodologists discuss weaknesses today. History, instrumentation, mortality, regression, and testing. The next section takes up selection, maturation, and the interactions among all of them.

Threat One

History

The history threat refers to events occurring outside the study. During the period between your measurements. Events that affect the dependent variable independently of anything you did. Picture a study of attitudes toward vaccination. Halfway through, a major news event about vaccine safety breaks. The apparent effect of your intervention may reflect the news, not the manipulation. Long intervals between pre-test and post-test, which give the world more time to interfere. Single-group designs, which have no comparison condition at all. And field settings, where outside events simply cannot be screened out. The standard defense is a control group living through the same period. If the historical event affects both groups equally, the comparison is preserved. But history can survive even into experimental designs. If a fire alarm interrupts the treatment session and not the control session, the event was not shared.

Threat Two

Instrumentation

Instrumentation threats arise when the measuring instrument itself changes. The change can be mechanical. Equipment drift, software updates, or calibration errors. Or it can be human, when raters and coders shift their judgments over time. A coder who becomes more lenient through fatigue produces drifting ratings. So does one who becomes stricter through experience. In both cases the differences across time reflect the rater, not the participants. These threats are especially common in long studies. Also in studies that switch instruments mid-stream, and in observational research. The defenses are practical. Use the same instrument throughout, and calibrate regularly. Train raters thoroughly, and re-check their consistency periodically. Where possible, blind raters to the time-point or condition of the data they are coding. And when an instrument change is unavoidable, there is a clever fix. Administer both old and new versions to a subset, and estimate the shift directly.

Threat Three

Mortality, or Attrition

Mortality is the third threat, and the label is unfortunate. It rarely refers to literal death. It means participants dropping out before completion. Here is the crucial refinement, and it is easy to miss. The threat to internal validity is not attrition itself. It is differential attrition. That means participants leaving in one condition at different rates, or for different reasons. Consider a demanding treatment that causes less-committed participants to drop out. The people remaining for the post-test are now more committed than the control group. So any apparent treatment effect may reflect that differential survival. Your treatment did not raise their scores. It removed the low scorers. Reduce attrition with realistic burden, follow-up contacts, and compensation. Track who drops out, and analyze whether they differ from those who completed. And report an intent-to-treat analysis, which retains all originally enrolled participants.

A Counter-Intuitive Case

Why 30% Can Be Fine and 15% Fatal

There is a genuinely counter-intuitive consequence of that distinction. A study can show thirty percent overall attrition and have no problem at all. That is an unhappy figure, but a familiar one in applied research. It is harmless if the people who left are similar in both conditions. The same study can show fifteen percent attrition and be seriously compromised. That happens if those participants all came from one condition. Or if they differed systematically from the people who remained. So the headline percentage tells you very little on its own. Which is why researchers should report attrition by condition, not just overall. And should compare the baseline characteristics of completers against dropouts. When you read a paper, look for both of those. Their absence is itself informative.

Threat Four

Regression to the Mean

Regression to the mean is the fourth threat, and it is different in kind. It is a statistical phenomenon, not a research artifact in the ordinary sense. Nobody made a mistake. It happens anyway. And it produces patterns in data that are easily mistaken for treatment effects. Whenever participants are selected because they scored extremely high or extremely low. Their scores on a later administration will tend, on average, to drift toward the population mean. The reason comes straight from Classical Test Theory. An extreme score reflects two things at once. The participant's true position, and a component of random error. To score at the very bottom, you usually need both a low true score and bad luck. On a second occasion, that random component will on average be smaller. Which pulls the score back toward typical, all by itself.

Threat Four

Exactly Where Regression Bites

Now notice exactly where this threat bites. It is most acute in studies that select participants on extreme initial scores. Which is to say, in most applied intervention research. A remediation program for low-performing students. An intervention for highly anxious patients. A training program for under-achieving sales staff. So without a comparison group, these participants will appear to improve on follow-up. Even if the intervention has no effect whatsoever. The remedy is a comparison group selected the same way. Measured on the same schedule, so regression is held constant across conditions. Both groups then regress equally, and the difference between them becomes meaningful again.

Threat Five

Testing Effects

The fifth threat is testing. It refers to the influence of taking a test on later performance on the same test. The effect can be fairly benign. Participants become familiar with the format, and simply answer faster. Or it can be substantive, which is more serious. They remember specific items. They learn from feedback. They rehearse strategies between the two administrations. In every case, the post-test score now contains something extra. Something beyond the construct of interest. Namely, the experience of having taken the pre-test. And that extra component is not distributed by chance. Everyone got it.

Two Sub-Types

Practice and Sensitization

Two sub-types of testing effect are worth distinguishing. Practice effects are improvements due to familiarity with the test format. Or to the simple act of having performed similar items before. Sensitization is subtler, and more interesting. Completing the pre-test alerts participants to the constructs the study cares about. That leads them to attend more carefully to relevant cues during the manipulation. Take a pre-test on attitudes toward charity. It may make participants more receptive to a subsequent charity appeal. More receptive than they would otherwise have been. So the pre-test did not merely measure the manipulation. It primed it. Which means the effect you observe may not occur in anyone who was not pre-tested.

Three Fixes

Designs That Manage Testing Effects

Three design choices reduce testing effects, and they address different parts of the problem. The first is to use parallel forms. Different versions of the test that measure the same construct. One for the pre-test, another for the post-test. That removes the memory-of-items component, while leaving the practice component in place. The second is to include a no-treatment control group. It takes the same pre-test and post-test as everybody else. So any testing effect is held constant across conditions, the familiar solution. The third is used when sensitization is the main concern. It is the Solomon four-group design. It crosses the presence or absence of a pre-test with the presence or absence of the manipulation. That lets researchers estimate the testing effect directly, rather than merely balancing it.

All Five

Mechanisms and Their Defenses

History. Outside events affect outcomes between your measurements. Defended by an equivalent control group living through the same time period. Instrumentation. The measurement procedure or the rater drifts during the study. Defended by stable instruments, calibration, and blinded, trained coders. Mortality. Differential attrition leaves you with non-equivalent groups. Defended by reducing burden, reporting attrition by condition, and intent-to-treat analysis. Regression. Extreme scores drift toward the average on retest. Defended by a comparison group selected on the very same criterion. Testing. Pre-test exposure changes post-test performance. Defended by parallel forms, a no-treatment control, or the Solomon four-group design. Now read down that third column and notice the pattern. Almost every defense is a comparison group, designed to hold the threat constant. That single idea does most of the work in the entire catalog.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Internal Threats: History, Instrumentation, Mortality, Regression, and Testing

What is the history threat to internal validity?
The history threat refers to events outside a study, occurring between measurements, that affect the outcome independently of the manipulation. For example, a major news story about vaccine safety breaking midway through a study of vaccination attitudes could shift attitudes on its own. Single-group designs, long pretest-posttest intervals, and field settings are most vulnerable. The standard defense is a control group that lives through the same period, so both groups are equally affected.
What is regression to the mean in research?
Regression to the mean is the statistical tendency for people selected because of extreme scores to score closer to the average when retested. An extreme score combines a person's true level with random error, and that error is usually smaller next time. So the lowest-scoring students chosen for tutoring will tend to improve even if the tutoring does nothing. A comparison group selected on the same criterion holds regression constant.
What is differential attrition and why does it matter?
Differential attrition is dropout that differs between conditions in rate or in the kind of people who leave, and it is the real threat behind the label mortality. Overall attrition alone is not the problem: a study with 30 percent dropout can be sound if leavers are similar across groups, while 15 percent dropout concentrated in one condition can seriously bias results. Report attrition by condition and use an intent-to-treat analysis.
What are testing effects and how can they be controlled?
Testing effects occur when taking a pretest changes performance on a later test. Practice effects come from familiarity with the format or items, while sensitization occurs when the pretest alerts participants to the constructs being studied and changes how they respond to the manipulation. Parallel forms remove memory for specific items, a no-treatment control group holds testing effects constant, and the Solomon four-group design estimates the pretest's effect directly.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →