External Threats: Sample and Stimulus Considerations

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 11.3 Free with a free account

External Threats: Sample and Stimulus Considerations

Research Methods in Psychology · Chapter 11: External Validity and GeneralizationSection 11.3 · lesson 67 of 9411 min · free with a free account

← Why Generalization Matters in External ValidityNext lesson: External Threats: Novelty, Reactivity, and Experimental Settings →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

Why the people studied and the materials they encounter limit how far a finding travels.

  • Identify the principal sample-related threats
  • Explain the WEIRD-sample critique and its scope
  • Define stimulus sampling and why it is overlooked
  • Apply diversification strategies to both

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,421 words). Read along, or search it.

External Validity and Generalization

External Threats: Sample and Stimulus Considerations

Nobody would claim a finding holds for all people based on twenty undergraduates. And yet researchers routinely claim a finding holds for a whole category of stimuli. Based on five sentences, or eight images. This section covers both halves of that problem. You will identify the principal sample-related threats to external validity. You will meet the WEIRD-sample critique, and see exactly what it does and does not claim. You will define stimulus sampling, which is the far more neglected half. And you will learn diversification strategies that reduce both kinds of threat.

Two Constraints

The Sample of People and the Sample of Materials

Threats to external validity arise from many sources. But two are common and consequential enough to deserve focused treatment. Features of the sample of participants. And features of the stimuli or materials those participants encounter. Both are typically chosen for practical convenience. Rather than for the breadth of generalization they would support. The result is a literature with a specific character. Many findings have narrower applicability than their authors realize. And not because the underlying effects are fragile. Because the studies sampled too narrowly along dimensions that mattered.

Sample Threat One

Convenience Sampling

It draws participants from groups that happen to be readily accessible. Introductory psychology classes. Online crowdsourcing platforms. Hospital patient registers, or the researcher's own personal networks. It has been the dominant mode of recruitment in psychology for decades. And its virtues are genuinely real, which is worth acknowledging. Low cost. Fast turnaround. And feasibility for the substantial samples that statistical analysis requires. But its limitation is equally real. People in convenient samples differ systematically from the general target population. On many characteristics at once, most of them unmeasured. So findings established this way may not extend to the populations researchers wish to describe.

Sample Threat Two

Volunteer Bias

The second sample threat operates inside the accessible population. Participants who agree to take part are not a random subset of those approached. Volunteer bias names the systematic differences between those who consent and those who decline. And the differences run in a fairly consistent direction. Volunteers tend, on average, to be more curious. More educated, more sociable. And more interested in the topic of the research than non-volunteers are. It matters when the dependent variable is correlated with those characteristics. In that case, the achieved sample produces an effect estimate that will not travel. Not straightforwardly, at least, to less self-selected groups.

Sample Threat Three

The WEIRD-Sample Critique

A large proportion of published findings rest on samples from WEIRD populations. Western, Educated, Industrialized, Rich, and Democratic. The critique makes two observations, and both matter. First, that WEIRD samples are psychologically unusual in many ways. In visual perception. In moral reasoning. In social comparison. In conceptions of the self, and in motivation, among others. Second, that they constitute a small minority of the world's population. So findings established only in WEIRD samples are poor candidates for broad claims. Claims about humans in general, or about cognition in general. But be careful about what the critique actually says. It does not invalidate WEIRD-sample research. It places clear limits on the kinds of theoretical claims such research can support.

A Bad Defense

Why University Students Are Not a Neutral Sample

The argument runs that university students are unproblematic because they are people too. And any human cognitive or behavioral mechanism should appear in them. That argument is too quick, and here is why. Even when an effect is genuinely a feature of human cognition, its magnitude may differ. So may its shape, across different populations. And magnitude often matters more than mere presence. For theoretical claims, and for applied ones especially. University students differ from older adults in working-memory capacity. They differ from less-educated samples in test-taking strategies. From non-Western samples in self-construal. And from clinical populations in basic functioning. Each of those differences can shape the size or form of the effect being studied. Even when the underlying mechanism really is shared across all of them.

Sample Threat Four

Recruitment Channels

The fourth sample threat is about how you reach people in the first place. How participants are reached shapes who ends up in the sample. Post a study on a single online platform, and you reach users of that platform. Recruit through one clinic, and you reach the patients of that clinic. Advertise through a particular community organization, and you reach its members. Each channel introduces selection on the characteristics that channel itself favors. The remedy is not to abandon convenience sampling entirely. That would be impractical, and nobody is proposing it. The remedy is to diversify recruitment across multiple channels. And then to report the channels used, so readers can assess the resulting sample.

The Overlooked Half

The Stimulus-Sampling Problem

If samples of people limit population generalization, samples of stimuli limit something else. The words, images, scenarios, items, and materials participants encounter. Those limit operations generalization. The threat is structurally identical to a sample-related threat. But it is very much more often overlooked. Consider the double standard, stated plainly. Researchers would never claim a finding holds for all people based on twenty undergraduates. Yet they routinely claim a finding holds for a whole stimulus category. Based on five sentences, or eight images. The logic to hold onto is simple. An effect generalizes to a construct only insofar as your stimulus set represents that construct.

Three Explanations

Why Limited Stimulus Sets Are Hazardous

This has a name. Stimulus sampling. It is sometimes called the language-as-fixed-effect problem, or stimulus-as-fixed-effect. It describes the error of treating the specific stimuli used as the entire category. The category the study claims to address. Demonstrate a memory effect using ten particular sentences, and you have a finding. Whether it extends to sentences in general depends on whether those ten were representative. There are three possible reasons a small stimulus set produced your result. First, the construct of interest genuinely produces the effect. In which case different stimulus sets would produce similar results. Second, idiosyncratic features of these particular stimuli produced it. Their length, their frequency, their emotional connotation, their visual properties. Third, both operate together, with each accounting for part of what you saw. And without varying the stimuli, you cannot tell which case is in play.

Two Remedies

Random Factors and Stimulus Norming

The first is statistical. Treat stimuli as a random factor alongside participants. Mixed-effects models partition variance across both participants and stimuli. That generates effect estimates holding across the stimulus categories sampled. Rather than across the specific items you happened to choose. The technique requires more stimuli than older fixed-effect approaches. And it is more demanding statistically, so it is not free. What it buys is inferences that are honest about the operations-generalization question. The second remedy is stimulus norming, and it happens before data collection. You pilot-test materials with separate samples. Verifying that they elicit the intended interpretation, and differentiate the conditions. And that they contain no irrelevant features that would confound the manipulation. A scenario meant to elicit moral disapproval should be shown to do that first. A face manipulation meant to convey trustworthiness should be rated on trustworthiness. And on plausible confounds too, such as attractiveness or perceived age.

The Defenses

Diversify, and Report the Basis of Selection

The defenses against both families follow one principle. Increase the diversity of what you sample, and report the basis of selection transparently. Against convenience and volunteer bias, diversify recruitment across channels. Recruit outside university populations where the research question warrants it. And track and report participant characteristics in detail. Against WEIRD over-reliance, plan replications across cultures or subcultures. Particularly where the theoretical claim is broad. Treat single-population findings as starting points, rather than as conclusions. Against limited stimulus sets, use more stimuli per condition than tradition demands. Treat stimuli as random factors in the analysis, and norm materials before deployment. Against construction artifacts in custom stimuli, pilot-test with separate samples. And check for confounds beyond the intended manipulation, reporting the results. And against recruitment-channel selection, use multiple channels and compare across them.

The Stance

The Reflective Sampler

Finish with a stance you can adopt when planning any study. It has three elements, and they run in order. First, identify explicitly what you intend the finding to generalize to. Which populations, which settings, which operationalizations. Second, ask whether the planned sample of participants and stimuli represents those targets. And consult prior literature on how the relevant populations and stimulus classes actually vary. Third, where the planned sample is more limited than the target, document the gap. And it almost always will be more limited. That is the normal case, not a failure. Then frame the study's contribution accordingly. Here is the part worth remembering. A study claiming only to establish a phenomenon in a specific population is not a weak study. Not if it says so, and expects further work to examine generalizability. That is a more honest and more useful contribution than a study that overclaims.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About External Threats: Sample and Stimulus Considerations

How does the recruitment channel affect external validity?
The channel used to recruit participants shapes who ends up in the sample, and so limits whom the findings describe. A study posted on one online platform reaches that platform's users, recruiting through one clinic reaches that clinic's patients, and advertising through a community organization reaches its members. Each channel selects for the characteristics it favors. Since abandoning convenient channels is rarely practical, researchers recruit through several channels, compare across them, and report which ones they used.
What is volunteer bias?
Volunteer bias is the systematic difference between people who agree to take part in research and those who decline. Volunteers tend, on average, to be more curious, more educated, more sociable, and more interested in the study topic. When the outcome is related to those traits, effects found in volunteers may not carry over to less self-selected groups. Recruiting through several channels and reporting participant characteristics in detail help readers judge the gap.
What is stimulus sampling in psychology?
Stimulus sampling means treating the specific stimuli in a study as a sample from a broader category rather than as the category itself. A memory effect shown with ten particular sentences generalizes to sentences in general only if those ten are representative. Treating stimuli as fixed is known as the language-as-fixed-effect problem. Using more stimuli, norming them in advance, and modeling them as a random factor in mixed-effects models address it.
Is it a problem to use college students as research participants?
It can be, because university students differ from other groups in ways that may change the size or form of an effect, even when the underlying mechanism is shared. Students differ from older adults in working-memory capacity, from less-educated groups in test-taking strategies, from non-Western samples in self-construal, and from clinical populations in basic functioning. Student findings are best treated as starting points until replicated in other populations.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →