Testing Hypotheses and Establishing Statistical Significance

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 15.2 Free with a free account

Testing Hypotheses and Establishing Statistical Significance

Research Methods in Psychology · Chapter 15: Inferential Statistics and Hypothesis TestingSection 15.2 · lesson 88 of 949 min · free with a free account

← The Building Blocks of Inferential StatisticsNext lesson: Deciding Between One-Tailed and Two-Tailed Tests →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

The procedure that turns sample data into a decision about a claim.

  • State null and alternative hypotheses for a question
  • Describe the logic of significance testing in five steps
  • Distinguish Type I from Type II errors
  • Interpret a p-value correctly

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,371 words). Read along, or search it.

Inferential Statistics and Hypothesis Testing

Testing Hypotheses and Establishing Significance

This is the procedure that decides what gets published in psychology. It is also the procedure most often misunderstood, including by people who use it daily. By the end of this section you will state null and alternative hypotheses for a research question. You will describe the logic of null hypothesis significance testing, in five steps. You will distinguish Type one from Type two errors. And explain the role of alpha and beta in each. You will interpret a p value correctly, and recognize the common misinterpretations. And you will explain why statistical significance and practical importance are not the same thing.

Indirect by Design

The Logic of Hypothesis Testing

Null hypothesis significance testing, often shortened to N H S T. It is the dominant framework for inferential statistics in psychology. And in most empirical sciences. Its logic is indirect, and that indirectness is what gives learners trouble. You would expect it to ask whether the research hypothesis is true. It does not do that. It never does that. Instead it assumes the opposite, which is called the null hypothesis. Then it asks how often data as extreme as yours would occur, if the null were correct. If such data would be very rare under the null, the researcher rejects the null. Rejects it in favor of the alternative. And if such data would not be rare, the null is retained.

An Analogy

The Courtroom

The standard analogy for this is a courtroom, and it is a good one. The defendant is the null hypothesis. The defendant is presumed innocent, until evidence against them becomes compelling. Notice that the court never proves innocence. It only fails to prove guilt. And two different errors are possible. Convicting an innocent defendant is one error. Failing to convict a guilty one is a different error entirely. Hypothesis testing operates on similar principles. The burden of proof is placed on the data. And there is a structured procedure for translating that proof into a decision.

The Procedure

The Five Steps of NHST

That procedure has five steps, and the order is not negotiable. First, state the hypotheses. Specify the null and the alternative before collecting or examining any data. Second, choose a significance level. That is alpha, the probability threshold below which the null will be rejected. The conventional choice is an alpha of point zero five. Third, identify the appropriate test. Select the test statistic suited to your design and data type. A z, a t, an F, a chi squared, or another. Fourth, compute the test statistic and the p value. That is the probability of observing data this extreme under the null. Fifth, make a decision, and interpret it in research context.

The Pair

Null and Alternative

Look more closely at the pair of hypotheses in step one. The null hypothesis, written H nought, is a precise claim about a population parameter. Typically that there is no effect, no difference, or no relationship. For a study comparing two means, the null asserts that the two population means are equal. Mu one equals mu two. The alternative hypothesis, written H one, is the claim you actually want to support. Typically that there is a difference, so mu one does not equal mu two. And the pair has a specific logical relationship. They are mutually exclusive, and exhaustive. Between them they cover every possible state of the world. Which is exactly what makes rejecting one count as support for the other.

Before the Data

Why the Timing Is Methodological

Stating those hypotheses before any data are examined is methodologically central. Not a formality. Central. Think about what changes when the order is reversed. Hypotheses crafted after seeing the results have lost the protection they were meant to provide. They no longer carry the same epistemic weight. Even when they are stated in exactly the same words. A prediction that was free to change to match the data predicted nothing. That is one reason pre-registration has become increasingly common. You publicly commit to hypotheses and analytic procedures before collecting data. It costs nothing, and it makes the commitment verifiable.

The Reference

The Null Distribution

Once the null hypothesis is in place, you can derive the null distribution. That is the distribution of the test statistic across all possible samples. All possible samples, if the null were true. It is a picture of a world where nothing is happening. Against that picture, you can locate the result you actually got. The p value is the probability, under the null distribution, of a statistic as extreme as yours. As extreme, or more extreme. A small p value indicates the observed data would be rare under the null. A large p value indicates the data are consistent with what the null predicts. Notice that consistent with is not the same as evidence for.

The Critical Distinction

What the p-Value Actually Means

Now the single most consequential point in this section. The p value is the probability of the data, given the null hypothesis. The probability of the data, given the hypothesis. It is not the probability that the null hypothesis is true, given the data. Those two quantities sound almost identical, and they are entirely different. Reversing a conditional probability changes what it means. Three more things the p value is not. It is not the probability that the result was due to chance. It is not the strength of the effect. And it is not the importance of the finding. Reading p values correctly is one of the most consequential literacy skills in modern science.

Two Errors

Type I and Type II

Hypothesis testing can produce two distinct kinds of error. A Type one error occurs when you reject a null hypothesis that is in fact true. You declare that an effect exists when it does not. The probability of a Type one error is alpha. The significance level you set in advance. A Type two error occurs when you fail to reject a null hypothesis that is in fact false. You miss an effect that really does exist. The probability of a Type two error is beta. And beta depends on sample size, on effect size, and on the alpha you chose. Notice the asymmetry there. You set alpha directly. Beta follows from the design.

Four Outcomes

The Decision Table

Those errors sit inside a table of four possible outcomes. The rows are your decision. The columns are the truth you cannot see. Suppose you reject the null, and the null is actually true. That is a Type one error, with probability alpha. Suppose you reject the null, and the null is actually false. That is a correct decision, with probability one minus beta, which is power. Now suppose you fail to reject, and the null is actually true. That is also a correct decision, with probability one minus alpha. And if you fail to reject when the null is false, that is a Type two error. With probability beta. Two correct outcomes, and two errors. Every study lands in one of those four cells.

The Complement

Statistical Power

One cell in that table has a name of its own. One minus beta, the complement of the Type two error rate, is statistical power. It is the probability of correctly rejecting a false null. In other words, the probability of finding an effect that is really there. High quality research aims for power of at least point eight zero. That means at least an eighty percent chance of detecting the effect of interest, if it exists. Low power means real effects are likely to be missed. And the consequences reach beyond the individual study. The field has increasingly recognized that underpowered studies waste resources. They produce unreliable literatures, even when individual results are reported correctly.

Significance Is Not Importance

What a Significant Result Says

Finally, the phrase that carries more weight than it can bear. A statistically significant result is one whose p value falls below alpha. That is the whole meaning of the phrase. It says the data would be unusual under the null hypothesis, and nothing more. It does not say the effect is large. It does not say the finding is theoretically important. And it does not say the result will replicate. With sufficiently large samples, even tiny effects become statistically significant. With small samples, even substantial effects may not reach significance. So a complete report includes five things. The test statistic, its degrees of freedom, the p value, an effect size, and an interpretation. The p value alone hides whether the effect matters. The effect size alone hides how reliable it is.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Testing Hypotheses and Establishing Statistical Significance

What are the steps of null hypothesis significance testing?
Null hypothesis significance testing has five steps: state the null and alternative hypotheses before seeing the data, choose a significance level (alpha, conventionally .05), select a test statistic suited to the design, compute the statistic and its p value, and decide whether to reject the null and interpret the result in context. The null usually states no effect, such as mu1 = mu2, and the alternative states that an effect exists.
What does a p value mean?
A p value is the probability of getting a test statistic as extreme as the one observed, or more extreme, if the null hypothesis is true. It is not the probability that the null hypothesis is true, not the probability that the result was due to chance, and not a measure of effect size or importance. A small p value means the data would be unusual under the null.
What is the difference between a Type I and a Type II error?
A Type I error is rejecting a null hypothesis that is actually true, a false positive, and a Type II error is failing to reject a null hypothesis that is actually false, a missed effect. The Type I error rate is alpha, which the researcher sets in advance. The Type II error rate is beta, which depends on sample size, effect size, and alpha. In the courtroom analogy, they are convicting an innocent defendant and acquitting a guilty one.
What is statistical power and what level is considered adequate?
Statistical power is the probability of correctly rejecting a false null hypothesis, equal to 1 minus beta. Researchers usually aim for power of at least .80, an 80 percent chance of detecting an effect of the size of interest if it exists. Power rises with larger samples, larger effects, and a higher alpha. Underpowered studies often miss real effects and contribute to unreliable research literatures.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →