Acknowledging the Boundaries of Generalization, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 16.2 Free with a free account

Acknowledging the Boundaries of Generalization

Research Methods in Psychology · Chapter 16: Evaluating and Interpreting Research FindingsSection 16.2 · lesson 93 of 9410 min · free with a free account

← Strategies for Evaluating Research FindingsNext lesson: Interpreting Non-Significant Results in Research →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

Where the conclusions of a study end, and how to know.

  • Distinguish statistical from theoretical generalization
  • Identify the threats that limit external validity
  • Recognize common over-generalizations in reporting
  • Communicate a study's boundaries accurately

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,316 words). Read along, or search it.

Evaluating and Interpreting Research Findings

Acknowledging the Boundaries of Generalization

A study answers a question about the people who were actually in it. Everything beyond that is an argument, not a result. By the end of this section you will distinguish statistical from theoretical generalization. You will identify the threats to external validity that limit how widely findings extend. You will recognize common over-generalizations, in published research and in popular reporting. You will apply ecological validity considerations to laboratory findings. And you will communicate the boundaries of a study's conclusions accurately and honestly. That last skill is what separates a careful writer from an enthusiastic one.

The Problem

Exact Conditions, Extended Claims

Every research study is conducted under specific conditions. A specific sample, in a specific setting. Using specific procedures, at a specific point in time. And the results of the study describe those exact conditions exactly. Generalizing beyond those conditions is what makes research useful. It is also where most overreach happens. A study of undergraduates in a controlled laboratory says a great deal about undergraduates in laboratories. It may say very little about adults in workplaces. A finding from one cultural context may not transfer to another. And an effect that appeared in two thousand three may have weakened or vanished by now. Careful research consumption means asking, deliberately, how far a conclusion can responsibly extend.

Layer One

Sample to Population

Generalization happens in three layers, each wider than the last. The narrowest extends from the sample studied to the population it was drawn from. This is the domain of inferential statistics, and it is the best supported layer. If the sample was selected randomly, statistical procedures give a principled basis for inference. Notice what does the justifying there. The sampling procedure itself, not the size of the sample or the sophistication of the analysis. Random sampling is the strongest support for sample to population generalization. Convenience samples provide much weaker grounds. Undergraduates available for course credit, or online panels recruited through advertisements. And the weakness remains even when the same statistical procedures are applied.

Layer Two

Studied Population to Other Populations

The second layer extends from the population that was studied to populations that were not. Take a study of stress responses in American college students. Does it say something about stress responses in Japanese factory workers? In retired Italian farmers? In seven year old children? Maybe. But the original sampling procedure cannot tell you. It sampled only the population that was actually studied. So justification has to come from somewhere else. From theoretical reasoning about why the effect should extend. From prior evidence in similar populations. Or from explicit replications. And in the absence of any of that, claims should be cautious. Phrased as hypotheses for future research, rather than conclusions of this one.

Layer Three

Studied Conditions to Other Conditions

The widest layer extends from the conditions of the study to other conditions entirely. Laboratory experiments offer tight control over the manipulation. But they use procedures that may not match real world settings at all. A finding about decision making under controlled time pressure sounds applicable. It may say much less than it appears about decision making in actual emergencies. A study of memory for word lists in quiet booths is another case. It may not generalize to memory for conversations in noisy meetings. The relevant criterion here has a name. Ecological validity. It is the degree to which research procedures resemble real world conditions. And here is the honest admission that follows. Most laboratory research has only modest ecological validity, by design.

Not Equal Support

Each Layer Rests on Something Different

Notice that those three layers are not equally well supported. The sample to population layer is supported by the sampling procedure. That is a formal warrant. You can check whether it applies. The between population layer is supported only by theoretical argument. Or by additional evidence from other studies. And the lab to real world layer is supported only by ecological validity considerations. Or by applied replications in the setting of interest. So a single paper's conclusions rest on progressively thinner ground as they widen. Stating these boundaries explicitly is part of honest research. Eliding them is one of the most common forms of overreach.

Threats

Six Things That Limit Generalization

Six specific threats limit generalization, and it helps to have names for them. Sample composition. Findings from one age, gender, ethnic, or educational group may not extend. Setting specificity. Effects seen in a laboratory may not appear in workplaces, schools, or homes. Temporal validity. Effects from one historical period may weaken, or even reverse, as society changes. Operationalization. A specific measure may not generalize to other ways of measuring the same thing. Cultural context. Effects in one cultural setting may not hold in another. Different values, different norms, different histories. And selection effects. Volunteers, opt-ins, and convenience samples differ from the broader population. They differ in ways that can affect the result itself. Each of these narrows the range of situations to which a finding applies.

Two Kinds

Statistical Versus Theoretical Generalization

Two kinds of generalization work very differently, and are worth separating. Statistical generalization extends a finding to other instances from the same population. Instances drawn by the same procedure. That is the kind supported by inferential statistics. Theoretical generalization is different. It extends a finding to other situations, on the basis of reasoning. Reasoning about why the effect should occur in the first place. Here is an example that makes the difference concrete. A laboratory demonstration shows that loud noise impairs concentration. That generalizes theoretically to other situations involving noise and concentration. Even though those situations were never statistically sampled.

The Weaker Warrant

Where Theoretical Generalization Gets Its Strength

But theoretical generalization does not have a formal warrant. Not the way statistical generalization does. There is no procedure you can point to and say, this is why it holds. Its strength depends on two things instead. The strength of the underlying theory. And the resemblance between the studied case and the new one. Which means it is arguable in a way statistical generalization is not. Statistical generalization can be checked against a procedure. Theoretical generalization has to be defended. And a reader who finds the resemblance thin can reasonably contest it.

Patterns to Recognize

Five Common Over-Generalizations

Five patterns of over-generalization appear repeatedly in reports and in coverage. The first is WEIRD samples, treated as if they were universal. The second is lab to life leaps. Effects from controlled conditions reported as if they applied directly in real settings. The third is single study extrapolation. One study's results presented as established fact, rather than provisional evidence. The fourth is population level claims from individual data. Findings about averages presented as descriptions of all individuals. An average difference between groups says almost nothing about any one person. And the fifth is outdated evidence. Findings from earlier decades treated as valid without checking for contemporary replication.

The WEIRD Problem

Which Humans Were Actually Studied

The first of those deserves a closer look, because it reshaped the field. An influential analysis of psychological research made a simple observation. The vast majority of empirical psychology has used a narrow slice of humanity. Samples drawn from Western, Educated, Industrialized, Rich, and Democratic societies. The acronym for that is WEIRD, which was very much the point. And yet findings were routinely reported as if they characterized humans in general. The analysis cataloged numerous phenomena that vary substantially across cultures. Cognitive, perceptual, and social phenomena alike. That challenged claims of universality the empirical base could not actually support. So reading psychology literature carefully today means building one habit. Of any claim about humans, ask which humans were actually studied.

Communicating

Boundaries Are Not Decorative

Finally, how boundaries should be communicated. A study's discussion section should state explicitly where the findings would generalize. And, just as importantly, where they would not. The limitations subsection should name the actual boundaries of the design. Sample restrictions, setting specificity, operationalization choices. Rather than treating them as decorative caveats that nobody reads. Press releases and media coverage should reflect those boundaries too. Rather than stripping them away in pursuit of a cleaner headline. And readers have a role here as well. Expect explicit boundary statements, and look for them. Their absence is itself a warning sign about the study you are reading.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Acknowledging the Boundaries of Generalization

What is the difference between statistical and theoretical generalization?
Statistical generalization extends a finding to other members of the same population sampled by the same procedure, while theoretical generalization extends it to new situations by reasoning about why the effect occurs. A lab finding that loud noise impairs concentration can be generalized theoretically to other noisy settings that were never sampled. Statistical generalization rests on a checkable sampling procedure; theoretical generalization has to be argued and can be contested.
What is ecological validity in psychology?
Ecological validity is the degree to which research procedures resemble real-world conditions. A study of memory for word lists in a quiet booth may not generalize to remembering conversations in noisy meetings, and decision making under lab time pressure may say less about real emergencies than it appears to. Most laboratory research has only modest ecological validity because it trades realism for control, so lab-to-life claims need applied replications or careful argument.
What does WEIRD mean in psychology research?
WEIRD stands for Western, Educated, Industrialized, Rich, and Democratic, describing the societies most psychology samples have come from. Joseph Henrich, Steven Heine, and Ara Norenzayan popularized the term in a 2010 paper showing that many cognitive, perceptual, and social phenomena vary across cultures. Findings from WEIRD samples were often presented as universal, so a careful reader asks which humans were actually studied before accepting a claim about people in general.
Can results from a convenience sample be generalized to a population?
Only weakly, because generalizing from a sample to its population is best supported by random sampling. Random selection gives statistical procedures a principled basis for inference, while convenience samples, such as undergraduates or online panels recruited through ads, provide much weaker grounds even when the same statistics are applied. Volunteers and opt-in participants can differ from the broader population in ways that affect the result itself.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →