External Threats: Novelty, Reactivity, and Experimental Settings

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 11.4 Free with a free account

External Threats: Novelty, Reactivity, and Experimental Settings

Research Methods in Psychology · Chapter 11: External Validity and GeneralizationSection 11.4 · lesson 68 of 9411 min · free with a free account

← External Threats: Sample and Stimulus ConsiderationsNext lesson: Practical Strategies to Boost External Validity →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

How the strangeness of a research situation produces effects that fade once it becomes ordinary.

  • Define novelty effects and why they inflate estimates
  • Explain reactivity as an external, not internal, threat
  • Identify multiple-treatment interference
  • Compare laboratory, field and online settings

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,437 words). Read along, or search it.

External Validity and Generalization

External Threats: Novelty, Reactivity, and Experimental Settings

A new teaching method produces impressive gains in its first year. By the third year, once it is simply the curriculum, the gains are gone. Nothing about the method changed. What changed was that it stopped being new. This section covers threats that come from the research situation itself. You will define novelty effects, and see why they inflate estimates. You will meet reactivity again, this time as a threat to external validity rather than internal. You will identify multiple-treatment interference in within-subjects designs. And you will compare laboratory, field, and online settings as sources of distinctive threats.

The Situation Itself

Why the Setting Is a Threat

The previous section examined the people and the materials in a study. This section turns to the situation itself. The bare fact of being in a research study. The novelty of the experimental procedure. And the features of the laboratory or field setting that distinguish it from everyday life. These threats are real, and they are often invisible to the investigators themselves. Here is why that happens. A researcher who has spent years inside laboratory settings stops noticing them. Stops noticing how unfamiliar those settings are to a typical participant. And how unrepresentative they are of the contexts where the behavior normally occurs.

Threat One

Novelty Effects

Novelty effects come first. They describe the tendency of a new procedure, intervention, or technology. A tendency to produce a stronger response than it will produce once the novelty wears off. A teaching method introduced as a special innovation may produce real gains. Gains that fade when the same method becomes ordinary curriculum. A new piece of workplace software may improve productivity initially. Through participant interest and engagement, rather than through anything the software does. And it loses those gains as it becomes routine. Novelty effects are particularly common in evaluations of interventions. Because the act of receiving something new is itself motivating. Entirely independent of whatever active ingredient the intervention contains.

Threat One

Two Ways Novelty Misleads, and Two Fixes

The threat to external validity here is twofold. First, magnitude. The effect estimated in the study may overstate what appears in routine practice. Second, duration, which is easier to miss. A study measuring outcomes during or shortly after the intervention captures the inflated effect. A follow-up months later may show the effect attenuated, or entirely absent. Two design features help separate novelty from genuine effect. The first is longer follow-up periods. Those let the novelty component dissipate, so any residual effect can be estimated separately. The second is an active control condition. A comparison group that receives something new, but theoretically inert. That holds the novelty component constant across conditions. So any difference reflects the specific ingredient, rather than novelty as such. Both features are common in well-designed clinical and educational trials, and uncommon elsewhere.

Threat Two

Four Reactivity-Related Threats

Four reactivity-related threats to external validity are worth naming together. The Hawthorne effect, where performance changes because of being observed. Pretest sensitization, where a pretest sensitizes participants to the manipulation. Demand characteristics, where cues from the study shape responses to the hypothesis. And multiple-treatment interference, where treatments interact within participants. You have met several of these before, in the internal-validity chapter. There, reactivity operated by shifting how participants respond to the manipulation. Here, the same phenomena operate in a genuinely different way. So the next few slides look at that difference carefully.

Threat Two

Reactivity to the Studied Context

As an external threat, reactivity does something specific. It ensures that the behavior observed in the study is not the behavior that occurs outside it. When participants know they are being observed, their behavior changes. Take a study of workplace communication. It is conducted with participants who know they are in a research project. So it is a study of communication-when-observed. It is not a study of communication in general. The magnitude and shape of the patterns you observe may not extend. Not to communication when nobody is watching. And the study itself contains no information about the size of that difference.

Threat Three

Pretest Sensitization, Again

Pretest sensitization appeared earlier as part of the testing threat. It has an important external-validity consequence too. Participants who complete a pretest before the manipulation become alert. Alert to the very constructs the study cares about. They may then respond to the manipulation more strongly. More strongly than participants who encounter it without that prior priming. Now look carefully at what survives and what does not. The within-study comparison may remain perfectly unbiased. Both conditions took the pretest, so the sensitization is shared. But the size of the effect may overstate what appears in an unsensitized population. The remedy, when feasible, is the Solomon four-group design. Or a separate post-test-only condition, letting you estimate the sensitization contribution.

The Key Distinction

Absolute Behavior Versus Relative Comparison

This is a subtle point, and it is where students most often lose the thread. So state the two questions side by side. Internal validity asks whether the manipulation produced the difference between conditions. Within the study, rather than something else. And reactivity that affects all conditions equally preserves that comparison completely. External validity asks a different question. Whether the same effect would appear outside the study. With people who are not aware of being studied. Who have not taken a pretest, and who are not embedded in a research situation. Here is the whole point in one line. Reactivity changes the absolute behavior, even when it does not change the relative comparison. And that matters whenever the size of the effect is part of what you want to estimate.

Threat Four

Multiple-Treatment Interference

The fourth threat belongs specifically to within-subjects designs. Those designs are internally efficient, because each participant serves as their own control. But they introduce a distinctive external-validity threat. Multiple-treatment interference. It describes the situation where earlier conditions shape responses to later ones. The effect of treatment B, encountered after treatment A, may not equal the effect of B alone. And this holds even when A's direct effect has completely dissipated. Because the participant has been changed by the experience of A. Now, counterbalancing was the standard remedy for order effects. It distributes them equally across conditions, which addresses the internal form. It does not address the external form at all. The average estimate still reflects responses given after other treatments. Rather than responses given in isolation.

Which Question Are You Asking?

Settings and Their Distinctive Threats

If the practical question is how a treatment performs as one of many similar experiences. In routine clinical care, for example. Then the within-subjects estimate may match practice rather well. If the question is how the treatment performs in isolation, use a between-subjects design. Different settings produce different threats, because they differ from everyday life differently. The laboratory's high control produces artificial conditions. Lacking the noise, the distractions, and the time pressures of natural settings. So effects may appear cleaner and larger in the lab than in life. In the field, reactivity to research presence may be lower. But uncontrolled context adds variability that can mask effects, or interact with the manipulation. Field studies often estimate smaller, noisier effects than laboratory studies do. And disagreement between lab and field need not mean that either one is wrong. In clinical and applied programs, staff allegiance and implementation variation dominate. Which is why effectiveness in routine practice often differs from efficacy in trials.

A Growing Modality

The Particular Case of Online Research

Online research deserves separate comment, because it has grown so rapidly. Large and demographically diverse samples. Low recruitment cost. And access to populations that would be hard to reach in person. The threats are also real, and considerably underappreciated. Participants on crowdsourcing platforms develop expertise as research participants. They become familiar with common paradigms, and learn what researchers expect. That is the professional participant phenomenon, and it constrains generalization. Attention is reduced, and competing distractions are entirely uncontrolled. Identity verification is limited. And platform-specific subcultures shape the norms of response. None of this invalidates online research, and it should not be read that way. It shapes what kinds of generalization online research can support. And it calls for design choices now standard in mature online programs. Attention checks, screening of repeat participants, and sensitivity analyses across platforms.

The Defenses

Replicate Across Settings

The most powerful defense against setting-related threats is replication across settings. A finding established in the laboratory should be examined in the field. Where the research question warrants that, at least. A finding established online should be examined in person. And a finding established in one cultural context should be examined in others. Particularly where the theoretical claim being made is broad. Within a single study, the available defenses are more modest. Longer follow-up, to allow novelty to fade before you measure. Active control conditions, which hold reactivity constant across groups. Between-subjects designs, where multiple-treatment interference is a real concern. And transparent reporting of the setting features that may have shaped your results. The final section brings all of these together as practical strategies.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About External Threats: Novelty, Reactivity, and Experimental Settings

What is the novelty effect in research?
The novelty effect is the tendency for a new procedure, intervention, or technology to produce a stronger response than it will once it becomes routine. A new teaching method may raise scores in its first year and lose those gains when it becomes ordinary curriculum. Novelty can inflate both the size and the apparent durability of an effect. Longer follow-up and an active control condition that is equally new help separate novelty from a genuine effect.
How does reactivity threaten external validity?
Reactivity threatens external validity because behavior observed in a study may differ from behavior when nobody is watching. A study of workplace communication among people who know they are being researched really measures communication-when-observed. Even when reactivity affects all conditions equally and the internal comparison stays fair, it can change absolute behavior and the size of the effect. Related threats include the Hawthorne effect, pretest sensitization, and demand characteristics.
What is multiple-treatment interference?
Multiple-treatment interference occurs in within-subjects designs when earlier conditions change how participants respond to later ones. The effect of treatment B given after treatment A may differ from the effect of B alone, even after A's direct effect has worn off. Counterbalancing spreads order effects evenly across conditions but does not solve this generalization problem. If the question is how a treatment works in isolation, a between-subjects design is the safer choice.
Are laboratory, field, or online studies more generalizable?
Each setting carries its own threats, so none is automatically more generalizable. Laboratory studies remove everyday noise, distraction, and time pressure, so effects can look cleaner and larger than in real life. Field studies may reduce reactivity but add uncontrolled variation that can mask effects. Online studies reach large, diverse samples but face reduced attention, limited identity checks, and professional participants familiar with common paradigms. Replicating across settings is the strongest defense.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →