Strategies for Evaluating Research Findings, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 16.1 Free with a free account

Strategies for Evaluating Research Findings

Research Methods in Psychology · Chapter 16: Evaluating and Interpreting Research FindingsSection 16.1 · lesson 92 of 949 min · free with a free account

← Comparing Multiple Groups with ANOVANext lesson: Acknowledging the Boundaries of Generalization →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

The questions a careful reader asks before believing a study.

  • Apply five complementary lenses to any finding
  • Separate statistical significance from practical importance
  • Use effect sizes to judge the magnitude of a result
  • Identify the warning signs that a finding may not replicate

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,292 words). Read along, or search it.

Evaluating and Interpreting Research Findings

Strategies for Evaluating Research Findings

You now know how research is done. This chapter is about deciding whether to believe it. By the end of this section you will apply five complementary lenses to any empirical finding. You will distinguish statistical significance from practical importance. You will recognize the common effect size measures. And use them to judge the magnitude of a result, rather than just its detectability. You will identify the warning signs that a finding may not replicate. And you will read a published study with a constructively critical eye. Constructively is the word doing the work there.

The Hardest Step

Why Evaluation Takes Longest to Learn

Here is something worth saying plainly at the start. The mechanics of research can be taught in a single course. How to conduct a study, compute a statistic, report a p value. The skill of evaluating what those results actually mean takes far longer. Practicing researchers keep refining it throughout their careers. The reason is that evaluation requires holding several questions in mind at once. Questions about chance, about magnitude, about reliability. About validity, and about real world significance. And recognizing that no single answer settles the matter. A study can be statistically significant and practically trivial. It can be theoretically important and empirically fragile. It can be brilliant in the laboratory and useless in the field.

Lens One

Statistical Significance

The first lens asks whether the result is more than chance. A statistically significant finding would be rare under the null hypothesis. Which suggests the observed pattern reflects something other than random fluctuation. That is a real piece of information, and it is worth having. But significance is necessary and far from sufficient. With sufficiently large samples, even tiny differences become significant. With small samples, even substantial differences may not reach significance. So the p value tells you how surprising the data would be under the null. It says nothing about how large the effect is. And nothing about how reliably it would appear in another study.

Lens Two

Effect Size

The second lens asks how large the effect is. Effect size measures express magnitude on a standardized scale. Crucially, independent of sample size. Cohen's d expresses the difference between two means in standard deviation units. By convention, small, medium, and large correspond to zero point two, zero point five, and zero point eight. Pearson's r expresses the strength of a correlation. There, small, medium, and large are point one zero, point three zero, and point five zero. Eta squared and partial eta squared serve similar purposes for analysis of variance. And odds ratios handle categorical outcomes, though they depend on baseline rates. Effect sizes belong alongside p values in any honest report. Because a result with p equals point zero three is a very different finding. Very different at a d of zero point one than at a d of zero point eight.

Lens Three

Replicability

The third lens asks whether the finding would hold up if the study were repeated. Single studies, however well conducted, are vulnerable. Vulnerable to chance, to sampling variation. And to idiosyncrasies of the particular conditions under which they were run. Credibility rises sharply when independent researchers obtain the same result. With different samples, and different procedures. This is not a hypothetical concern for psychology. The replication crisis emerged in the field during the twenty tens. It demonstrated that many widely cited findings did not replicate. And it prompted widespread reform in research practices.

Reading for Replication

Three Questions to Ask

So what does reading with replicability in mind actually look like? It means asking three questions. First, what have other studies found on the same topic? A finding does not arrive alone, and its neighbors matter. Second, is this result novel, or part of a converging body of evidence? Novelty is exciting. Convergence is convincing. Third, how robust does the finding appear? Robust across different operationalizations, and across different samples. A single striking result is usually worth less than a modest one seen three times. Seen in three laboratories, measured three different ways.

Lens Four

Validity

The fourth lens asks a question the statistics cannot answer. Did the study measure what it claimed to measure? And did it test what it claimed to test? Construct validity is undermined when a measure captures the wrong thing. When a happiness scale turns out to be a measure of social desirability, for example. The numbers are fine. The label on them is wrong. Internal validity is undermined when alternative explanations remain available. Confounds. Selection effects. Demand characteristics. Or measurement artifacts that could account for the pattern as well as the proposed cause. Here is the uncomfortable implication of this lens. A finding can be significant, large, and replicable, and still be wrong about what it means.

Lens Five

Practical Meaning

The fifth lens asks whether the finding matters in the world. A reliable, large, valid effect can still be practically trivial. Trivial if the context in which it operates is rare. Or if the cost of acting on it is prohibitive. Or if its real world implications are simply too small to justify attention. And the relationship runs the other way too. A small effect can be practically transformative. If it is applied to very many people, or in a high stakes domain. Or as one part of a cumulative intervention. Notice that practical meaning is not derivable from the statistics alone. It requires knowledge of the context in which the finding would be applied.

The Key Rule

The Lenses Are Complementary, Not Substitutes

Now the rule that ties the five lenses together. A common error is treating one lens as a replacement for the others. So state the independences explicitly. A significant p value does not establish a large effect. A large effect does not guarantee replication. Replication does not establish validity. And validity does not guarantee practical importance. Each of those is a separate question, with a separate answer. Each lens reveals something the others cannot. So a careful evaluation looks through all five before drawing conclusions.

Warning Signs

When to Look Harder

Some patterns in published research signal that a finding needs a closer look. Very small samples are the first. They are underpowered to detect real effects. And when they do reach significance, the false positive rate among those results is high. No effect size estimate reported is the second. The reader simply cannot judge the magnitude of what was found. Many comparisons without correction is the third. Type one error is inflated, so some of those findings are likely false positives. Fourth, hypotheses that appear to follow the data. Post hoc theorizing loses the protection of pre-specified predictions. Fifth, non-replication reported elsewhere, which suggests an outlier result. And sixth, a conflict between the effect size and the p value.

The Right Posture

Curiosity, Not Cynicism

One caution about how to hold all of this. Critical evaluation is sometimes confused with reflexive skepticism. Assuming every finding is wrong until proved otherwise. That stance is no more useful than naive credulity. Both of them replace judgment with a default. The goal of evaluation is not to dismiss research. It is to read it accurately. To distinguish strong findings from weak ones. Well designed studies from flawed ones. And theoretically important results from statistical curiosities. Apply the five lenses honestly, and some studies will survive scrutiny while others do not. That resulting calibration of confidence is what scientific literacy actually consists of.

A Signal Worth Reading

Good Studies Welcome the Lenses

Finally, a heuristic that is more reliable than it has any right to be. A well designed, well reported study welcomes the lenses in this section. Effect sizes are reported because the authors want readers to judge magnitude. Limitations are acknowledged because pretending otherwise weakens the case. Replication efforts are cited because converging evidence strengthens the claim. Good work invites the scrutiny, because scrutiny is what makes it credible. Now notice the inverse signal. Some studies hide from scrutiny instead. Through selective reporting, vague methods, or unjustified analytic choices. Studies that do that usually do it for a reason. Honest evaluation rewards honest research.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Strategies for Evaluating Research Findings

How do you evaluate whether a research finding is credible?
Evaluate a finding through five separate questions: whether it is more than chance, how large the effect is, whether it replicates, whether the study measured and tested what it claimed, and whether it matters in the real world. Each lens answers a different question. A significant result need not be large, a large effect need not replicate, and a replicated finding can still be invalid or practically trivial.
Does statistical significance mean a result is important?
No, statistical significance means the data would be unusual if the null hypothesis were true, not that the effect is large or important. With very large samples, tiny differences become significant, and with small samples, substantial differences may not. Importance depends on effect size and context: a small effect can matter greatly in a high-stakes area such as mortality or when applied to many people, while a large effect can be trivial in a rare situation.
What counts as a small, medium, or large effect size?
By Cohen's conventions, a d of 0.2 is small, 0.5 medium, and 0.8 large, where d expresses the difference between two means in standard deviation units. For Pearson's r, the benchmarks are .10, .30, and .50. These are rough guides rather than fixed rules. Reporting an effect size matters because p = .03 means something very different when d is 0.1 than when d is 0.8.
What are warning signs that a study may not replicate?
Warning signs include very small samples, no reported effect size, many comparisons without correction, hypotheses that seem written to fit the data, failed replications reported elsewhere, and an effect size that conflicts with the p value. Psychology's replication crisis in the 2010s showed that many widely cited findings did not replicate. Converging evidence from independent labs, different samples, and different measures is more convincing than one striking result.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →