Recognizing Common Sampling Challenges, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 8.4 Free with a free account

Recognizing Common Sampling Challenges

Research Methods in Psychology · Chapter 8: Sampling and GeneralizationSection 8.4 · lesson 48 of 9410 min · free with a free account

← Probability vs. Non-Probability Sampling ApproachesNext lesson: Determining the Appropriate Sample Size →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

Anticipating and managing the problems that arise when sampling meets the real world.

  • Separate undercoverage, non-response and self-selection
  • Recognize frame errors and who they affect
  • Apply mitigation strategies to each challenge
  • Report sample limitations transparently

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,393 words). Read along, or search it.

Sampling and Generalization

Recognizing Common Sampling Challenges

A sampling plan is a promise about who will end up in your data. Reality does not keep that promise. Frames turn out incomplete. People refuse. Lists go stale. Whole groups turn out to have been unreachable from the start. This section names the six challenges that show up most often. You will separate undercoverage from non-response, and both from self-selection. You will recognize frame errors, and who they quietly affect. You will learn what to do about each one. And you will learn how to report what you could not fix.

Plan Meets Reality

Six Challenges That Arrive Anyway

The sampling methods are usually described as if they worked cleanly. Probability sampling produces representative samples. Purposive sampling captures information-rich cases, and so on down the list. Real-world sampling rarely matches those idealized descriptions. Six challenges arise routinely, even in well-designed studies. Undercoverage, when the frame misses people it should have contained. Non-response, when the people you selected decline to take part. Self-selection, when participants choose themselves instead. Frame errors, where the entries themselves are wrong. The voluntary-response problem, a particularly severe form of self-selection. And systematic exclusion, where entire groups never had a chance of appearing.

Plan Meets Reality

Imperfection Is Not Invalidity

Sampling frames are incomplete. Selected respondents refuse to participate. Whoever volunteers differs systematically from those who do not. Lists are outdated. And whole populations are quietly excluded by the channels through which research recruits. Now here is the thing to be clear about. These problems do not invalidate research. Every study has some of them. What they require is explicit recognition. A sampling plan that ignores them produces overconfident conclusions. A plan that addresses them, through prevention, mitigation, or honest acknowledgment, produces credible work. Credible even when the conditions were imperfect, which they always are.

Challenge One

Undercoverage

Start with undercoverage. It occurs when the sampling frame fails to include all members of the target population. The classic cases involve telephones. A sample drawn from landline directories misses households without landlines. A sample drawn from mobile numbers misses those without phones. And an online panel misses anyone without reliable internet access. University student rolls miss those who recently enrolled, withdrew, or attend part-time. Voter registration lists miss non-citizens, unregistered citizens, and recent movers. But notice what actually determines whether a blind spot matters. The question is whether the people outside the frame differ from those inside it. And specifically on the variables you care about. If they do not differ, the gap is harmless. Usually they do differ.

Fixing Coverage

Three Ways to Close the Gap

Three approaches reduce undercoverage. The first is simply to use a better frame. One that more fully covers the target population you actually care about. The second is to combine frames. Multi-frame sampling merges two or more frames, so each one's blind spot gets covered. The third is to weight afterwards. Post-stratification weights adjust the data using known population characteristics. In that case, honest reporting limits the population to which findings apply. That is a real result, not a failure of nerve. If the study describes landline households, then say landline households.

Challenge Two

Non-Response and Its Bias

Non-response is the second challenge, and it has grown steadily worse. Non-response simply means that selected units fail to participate. Response rates have fallen sharply across many populations and modes. Sometimes below ten percent in modern telephone and online surveys. It is non-response bias. That is the possibility that those who respond differ systematically from those who do not. And that distinction leads to a genuinely counter-intuitive point. Higher response rates do not by themselves rule out bias. A ninety percent response rate can still be biased if the missing ten percent are distinctive. What very low response rates do is make bias more plausible. So a rate is evidence about risk. It is not a verdict on quality.

Fixing Response

Six Strategies for Non-Response

Six strategies reduce non-response, or adjust for it afterwards. Multiple contact attempts reach those who were simply hard to catch the first time. That is different from those who are refusing you. Modest incentives encourage participation among less-motivated respondents. Which helps balance a sample that would otherwise skew toward the enthusiastic. Shorter, well-designed instruments reduce refusal driven by perceived burden. Offering multiple modes, mail, phone, and online, lets people use their preferred channel. Non-response follow-up is cleverer than it sounds. You run brief surveys of non-respondents, to characterize how they differ. Then you can adjust the analysis with actual knowledge, rather than assumption. And statistical weighting adjusts results to known population proportions on key characteristics.

Challenge Three

Self-Selection Bias

Self-selection bias is the third challenge. It arises when participants choose themselves into the sample. Rather than being selected by the researcher. It is the defining feature of voluntary-response and online-poll samples. But here is what people miss. It infiltrates probability designs too. Suppose only a small fraction of your randomly selected individuals agree to take part. The sample you actually realized is effectively self-selected from the willing. And people who choose to participate typically differ from those who do not. They are often more interested in the topic, and more confident in their views. More available, and more trusting of researchers. Those differences produce estimates that look precise and mislead anyway. Think of it this way. Self-selected samples capture loud voices and miss quiet ones. A website feedback link hears from people moved to click. The data describe the speakers. They do not describe the population.

Challenge Four

Frame Errors

Frame errors are the fourth challenge, and the most mundane. Even when a frame nominally covers the target population, individual entries can be wrong. Outdated entries are people who have moved, changed numbers, or died. Those selected units simply cannot be contacted. Duplicates are the same individual listed more than once in the frame. That person then has a higher probability of selection than you intended. Which quietly breaks the known-probability requirement your statistics depend on. Ineligible entries are units that should not be in the frame at all. Businesses sitting on a residential list, for example. Those waste recruitment effort, and cause confusion if nobody screens them out. Fix all three with recently updated lists, screening at recruitment, and duplicate checks. And when the frame comes from administrative records, regular audits keep it usable.

Challenge Five

The Voluntary-Response Problem

The voluntary-response problem deserves its own name. It is a distinct and especially severe form of self-selection. It happens when researchers issue an open invitation to anybody. Click here to take our survey. Call this number to share your view. Then they analyze whoever happens to respond. The people who respond are typically those with the strongest feelings about the topic. Often disproportionately negative, or disproportionately positive. The moderate middle simply does not bother. Such samples can still have value for exploratory work or generating hypotheses. But they should never be presented as estimates of population views. No matter how large the response was. Treating an online voluntary poll as a probability sample is among the most common errors in popular reporting.

Challenge Six

Systematic Exclusions

The last challenge is the quietest, because nobody chooses it. Some groups are routinely excluded not by intent. They are excluded by the structure of the sampling channels themselves. People without stable housing lack listed addresses, and standard frames cannot reach them. Institutionalized populations fall outside most household-based frames. That means hospital, prison, and care-home residents. Linguistic minorities are omitted whenever surveys run only in dominant languages. The very young and the very old are often excluded for ethical or feasibility reasons. The digitally unconnected disappear as research moves online. And people with severe disabilities are missed when recruitment and instruments are not accessible. Notice that no researcher decided to exclude any of them. The method did it, silently, before anyone looked at the data.

The Stakes

Invisible Exclusions, Real Consequences

Follow one of those exclusions all the way through to policy. Suppose health surveys consistently exclude the institutionalized and the homeless. And those without phones. The population health picture they produce understates the burden of disease. And it understates it precisely in the groups that need the most attention. So recognizing who is excluded is part of interpreting what a sample can tell you. And part of knowing what it cannot. Every sample has limitations, even after your best efforts. It describes how the frames were constructed. It reports the response rate, and any evidence about non-response bias. It compares the composition of the realized sample with the target population. And it names the groups the methods were unable to reach. Honesty about limitations is not weakness. It is the foundation of trustworthy research.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Recognizing Common Sampling Challenges

What is undercoverage in sampling?
Undercoverage occurs when the sampling frame leaves out members of the target population. Examples include landline directories missing households without landlines, online panels missing people without reliable internet, and voter lists missing unregistered citizens and recent movers. It matters when excluded people differ on the variables being studied. Remedies include better frames, combining frames, post-stratification weighting, and limiting conclusions to the population actually covered.
Does a high survey response rate mean the results are unbiased?
No, a high response rate does not by itself rule out non-response bias. Bias depends on whether non-respondents differ systematically from respondents, so a 90 percent response rate can still mislead if the missing 10 percent are distinctive. Very low response rates make bias more plausible but do not prove it. Follow-up surveys of non-respondents and statistical weighting help assess and adjust for it.
What is voluntary response bias?
Voluntary response bias occurs when an open invitation, such as an online poll or call-in line, lets anyone respond and those with the strongest feelings answer most. Their views are often disproportionately negative or positive, while the moderate middle stays silent. Such samples can help with exploratory work or generating hypotheses, but they should not be presented as estimates of population views, however many people responded.
What are frame errors in sampling?
Frame errors are mistakes in the entries of a sampling frame, even when it nominally covers the target population. Outdated entries list people who moved, changed numbers, or died. Duplicates list someone more than once, raising their chance of selection and breaking the known-probability assumption. Ineligible entries, such as businesses on a residential list, waste effort. Updated lists, screening at recruitment, and duplicate checks reduce these errors.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →