Interpreting Non-Significant Results in Research, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 16.3 Free with a free account

Interpreting Non-Significant Results in Research

Research Methods in Psychology · Chapter 16: Evaluating and Interpreting Research FindingsSection 16.3 · lesson 94 of 949 min · free with a free account

← Acknowledging the Boundaries of Generalization

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

What a large p-value tells you, and the much longer list of what it does not.

  • Explain why non-significance does not establish the null
  • Name the three reasons a study can miss an effect
  • Distinguish absence of evidence from evidence of absence
  • Apply equivalence testing and Bayesian inference correctly

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,302 words). Read along, or search it.

Evaluating and Interpreting Research Findings

Interpreting Non-Significant Results

Your study came back non-significant. What have you actually learned? Much less than most people assume, and more than most people admit. By the end of this section you will explain why a non-significant result does not establish the null. You will identify the three principal reasons a study might fail to reach significance. You will distinguish absence of evidence from evidence of absence. You will recognize the alternative frameworks that can support a no effect conclusion. Equivalence testing, and Bayesian inference, when properly applied. And you will communicate null findings honestly, without overstating or dismissing them.

The Asymmetry

The Procedure Only Points One Way

Null hypothesis significance testing is structured asymmetrically. That asymmetry is deliberate, and it is the source of all the confusion here. The procedure is designed to give grounds for rejecting the null. Rejecting it when the data are sufficiently extreme. It is not designed to give grounds for accepting the null when the data are not extreme. A small p value licenses a directional conclusion. The observed data would be unusual under the null, so something else is operating. A large p value licenses no comparable conclusion at all. The data are consistent with the null. They are also consistent with many other possibilities. So treating a large p value as proof that the null is true reverses the asymmetry. The very asymmetry the whole procedure was built around.

The Phrase

Absence of Evidence Is Not Evidence of Absence

This logic gets summarized in a phrase that appears throughout statistical writing. Absence of evidence is not evidence of absence. Take that phrase apart, because it repays the attention. A study that fails to detect an effect tells you something, at most. That this particular study did not detect the effect. That is a fact about the study, not a fact about the world. It does not establish that the effect is absent. Those are two entirely different claims. And only one of them was ever actually tested.

Reason One

The Null Is Actually True

There are three distinct reasons a study might miss significance. Only one of them corresponds to a true null. The first possibility is that no real effect exists in the population. The manipulation does nothing. The variables are unrelated. The groups truly do not differ. A non-significant result is certainly consistent with that. But it is not the only state of affairs consistent with the result. And that distinction is the whole problem. Without additional evidence, the data cannot distinguish a true null from the alternatives.

Reason Two

The Effect Exists but Is Small

The second possibility is that an effect exists but is too small to detect. Too small for the study as designed, specifically. Recall what statistical power depends on. Sample size, effect size, and the chosen significance level. That means power is not a single property of a study. A study with adequate power for a large effect may have very low power for a small one. Same design, same participants, different question. So an effect that is real but modest can produce non-significant results repeatedly. Study after study, even when the underlying phenomenon is genuine. Underpowered studies are particularly vulnerable to this. With power below fifty percent, a real effect is more likely to be missed than detected.

Reason Three

The Design Obscured It

The third possibility is that an effect exists but the study could not capture it. Several things can produce that outcome. Measurement error, which adds noise around whatever signal is present. Ceiling or floor effects, where scores cannot move in the direction you predicted. Demand characteristics that worked against the manipulation. Confounds that pulled in the opposite direction and canceled the effect out. Or implementation failures that prevented the manipulation from taking hold at all. Any one of these produces a non-significant result even when the effect is real. Studies with weak manipulations or noisy measures are especially vulnerable. So evaluating a null result requires one question above all. Was this study well positioned to detect the effect of interest in the first place?

Two Practices

What Makes a Null Result Interpretable

Two practices substantially improve how interpretable a null result is. The first is conducting a power analysis before data collection. That ensures the study has a reasonable chance of detecting the effect, if it exists. Without that planning, a non-significant result is ambiguous in a specific way. It may indicate only that the design was inadequate. The second practice is pre-registering hypotheses and analytic procedures. That protects against a very human temptation. Recharacterizing non-significant predicted results as exploratory findings about something else. Both practices are increasingly standard in rigorous research. And both have to happen before the data arrive to be worth anything at all.

The Correction Table

What Non-Significance Does Not Mean

Five common readings of a null result are wrong. Take them one at a time. It does not mean the null hypothesis is true. The data are consistent with the null, and also with various alternatives. It does not mean the effect does not exist. This particular study did not detect it. The effect may still be perfectly real. It does not mean the hypothesis was wrong. The hypothesis received no support here. It has not been disproved. It does not mean the study was a failure. The result is informative, particularly when combined with other evidence. And it does not mean no further research is warranted. Replication, larger samples, or improved designs may yet reveal an effect.

A Real Need

When You Want to Argue for No Effect

Sometimes a researcher genuinely wants to argue that no effect exists. Not as a fallback, but as the actual goal of the study. That a treatment works no better than placebo. That two procedures are equivalent. That a predicted relationship does not, in fact, hold. Those are legitimate and often important research questions. And standard significance testing simply cannot support the conclusion. The framework has no mechanism for concluding in favor of the null. Two other analytic frameworks address that gap directly.

Framework One

Equivalence Testing

The first is equivalence testing, and it works by inversion. It flips the structure of standard hypothesis testing on its head. The researcher begins by specifying a region of negligible effect sizes. Small enough that, for theoretical or practical purposes, the effect counts as absent. Then the hypotheses swap places. The null becomes that the true effect lies outside this equivalence region. The alternative is that it lies within. So a significant result now licenses a very different conclusion. That the effect is too small to matter, given the bounds you specified. Equivalence testing is widely used in pharmaceutical research. There, showing that a generic drug performs equivalently to a brand name one is the explicit goal.

Framework Two

Bayesian Inference

The second framework is Bayesian inference. Bayesian methods compute the relative evidence in the data for both hypotheses. Often expressed as a Bayes factor. A Bayes factor of ten means the data are ten times as probable under the alternative. Ten times as probable as under the null, that is. A Bayes factor of one tenth means the reverse. And that is the crucial difference from standard testing. Bayesian comparison can produce evidence in either direction. Including evidence in favor of the null hypothesis. So a study that missed significance can be reanalyzed in that framework. One warning applies to both frameworks, and it is strict. Equivalence bounds and prior distributions must be specified before data collection.

Reporting

Four Practices, and the File Drawer

Finally, four practices for reporting null findings honestly. First, report effect sizes alongside p values. That lets readers see how large the observed effect was, even without significance. Second, report and inspect confidence intervals. A wide interval around zero tells a very different story than a narrow one. Even though both of them contain zero. Third, preserve the distinction in your writing. Say the study did not detect a difference, rather than it showed no difference. Fourth, publish the result rather than hiding it. The file drawer problem of unpublished null results distorts cumulative knowledge. Because a null result is genuinely informative to the field. It says that with this design, this sample, and this measure, the effect was not detected.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Interpreting Non-Significant Results in Research

Does a non-significant result mean there is no effect?
No, a non-significant result means the study did not detect an effect, not that the effect is absent. Significance testing is built to give grounds for rejecting the null, not for accepting it, and a large p value is consistent with the null and with many alternatives. Absence of evidence is not evidence of absence, so it is more accurate to write that a study did not detect a difference than that it showed none.
Why might a study fail to find a significant effect?
A study can miss significance for three reasons: no effect exists, a real effect is too small for the study's power, or a real effect exists but the study could not capture it. Capture can fail through measurement error, ceiling or floor effects, opposing confounds, demand characteristics, or a manipulation that never took hold. With power below 50 percent, a real effect is more likely to be missed than found.
What is equivalence testing?
Equivalence testing is a method for showing that an effect is too small to matter. The researcher sets bounds for a negligible effect size before collecting data, then reverses the usual hypotheses: the null says the true effect lies outside the bounds and the alternative says it lies inside them. A significant result supports equivalence. It is widely used in pharmaceutical research, for example to show that a generic drug performs equivalently to a brand-name drug.
How can a Bayes factor provide evidence for the null hypothesis?
A Bayes factor compares how probable the data are under two hypotheses, so it can favor either one, including the null. A Bayes factor of 10 means the data are ten times as probable under the alternative as under the null, while a Bayes factor of 1/10 means they are ten times as probable under the null. The prior distributions should be specified before data collection, not chosen after seeing the results.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →