Comparing Multiple Groups with ANOVA, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 15.5 Free with a free account

Comparing Multiple Groups with ANOVA

Research Methods in Psychology · Chapter 15: Inferential Statistics and Hypothesis TestingSection 15.5 · lesson 91 of 949 min · free with a free account

← A Closer Look at Chi-Square and t-TestsNext lesson: Strategies for Evaluating Research Findings →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

Why running many t-tests is the wrong solution, and what to do instead.

  • Explain how multiple t-tests inflate the Type I error rate
  • Describe the logic of the F-ratio
  • Read an ANOVA summary table
  • Apply post-hoc procedures after a significant omnibus test

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,337 words). Read along, or search it.

Inferential Statistics and Hypothesis Testing

Comparing Multiple Groups with ANOVA

You have three groups instead of two. The obvious move is also the wrong one. This section explains why, and what replaces it. By the end you will explain why multiple t tests inflate the Type one error rate. You will describe the logic of the F ratio in analysis of variance. You will distinguish one way from factorial designs. And between-subjects variants from repeated measures ones. You will interpret an ANOVA summary table, line by line. And you will apply the right post hoc procedure after a significant omnibus test.

The Temptation

One t-Test Per Pair

When a study compares more than two groups, one temptation is very natural. Run a t test on every pair. A versus B. A versus C. B versus C. And so on. The number of pairs grows quickly. Three groups yield three pairwise comparisons. Four groups yield six. Five groups yield ten. Now here is the problem with running all of them. Each individual t test carries a Type one error rate of alpha, typically point zero five. And those error rates accumulate across the whole set.

The Arithmetic

How Fast Protection Erodes

Put numbers on that accumulation, because they are worse than most people expect. With three groups and three comparisons, the false positive probability rises. The probability of at least one false positive somewhere is approximately point one four. With ten comparisons, it exceeds point four zero. Nearly a coin flip that something in your analysis is spurious. So the conventional five percent protection is silently abandoned. Abandoned the moment multiple tests are run without correction. And nothing in the software output warns you that it happened. Analysis of variance solves this, and everyone abbreviates it to ANOVA. It provides a single omnibus test that compares all groups simultaneously. While preserving the overall Type one error rate at alpha.

The Partition

Two Sources of Variability

The way ANOVA does that is by partitioning variability. It splits the total variability in a dataset into two components. Between group variability, and within group variability. Between group variability reflects differences among the group means. It includes any genuine effect of the manipulation. Plus some random fluctuation, because group means bounce around even with no effect. Within group variability reflects differences among observations inside each group. Those observations all received the same treatment, so any spread there is noise. That makes within group variability a baseline. It tells you how much variation occurs even when conditions are constant.

The Statistic

The F-Ratio

The F ratio compares those two quantities directly. It is between groups mean square, divided by within groups mean square. Mean square just means sum of squares divided by degrees of freedom. Now think about what the null hypothesis predicts for that ratio. If all group means are equal, between group variance reflects only random fluctuation. The same fluctuation that drives within group variance. So the two would be measuring the same thing. Which means the F ratio is expected to be approximately one. When the manipulation produces real differences, between group variance grows. Within group variance stays roughly constant, so the ratio rises above one. Sufficiently large values land in the rejection region of the F distribution.

The Table

An ANOVA Summary

ANOVA results are conventionally reported in a summary table. It presents the decomposition of variability transparently. Consider a three group study with ten participants per group. The between groups row shows a sum of squares of two hundred and forty. With two degrees of freedom, giving a mean square of one hundred and twenty. The within groups row shows a sum of squares of four hundred and five. With twenty seven degrees of freedom, giving a mean square of fifteen. Divide one hundred and twenty by fifteen, and you get an F of eight. With a p value of point zero zero one. And the total row shows six hundred and forty five, with twenty nine degrees of freedom. Notice that the two component rows add up to the total, in both columns.

Reading It

Where Every Number Comes From

Walk through where each column comes from. Sums of squares measure the total variability attributable to each source. That is a total, not an average, so it grows with sample size. Dividing sums of squares by degrees of freedom gives mean squares. A mean square is an average squared deviation, so it is comparable across sources. The F ratio is then between groups mean square over within groups mean square. And the p value says how unusual that F would be if the null were true. The degrees of freedom split in a specific way that is worth memorizing. The total is N minus one. Between groups gets k minus one, where k is the number of groups. And within groups gets N minus k.

Variant One and Two

One-Way and Repeated-Measures

ANOVA comes in several variants. Start with the two most common. The one way between subjects ANOVA is the variant in that summary table. It compares means across three or more independent groups. Groups defined by a single independent variable. Each participant contributes data to exactly one group. So it is the natural extension of the independent samples t test. The repeated measures ANOVA compares three or more conditions instead. Conditions experienced by the same participants. It exploits the pairing within participants to remove individual differences. Which increases power, exactly as the paired samples t test does for two conditions. It carries one extra assumption, called sphericity. When sphericity fails, corrections such as Greenhouse Geisser are applied.

Variants Three and Four

Factorial and Mixed

Two more variants complete the family. Factorial ANOVA extends the framework to two or more independent variables. In a fully crossed design, as you saw conceptually in the factorial chapter. It produces separate F tests for each main effect, and for each interaction. So the researcher can disentangle joint and independent contributions of the factors. The summary table grows correspondingly. It gains a row for each main effect and interaction, alongside the error term. Mixed ANOVA handles designs with at least one factor of each kind. One between subjects factor, and one within subjects factor. It combines features of between and within subjects tests in a single analysis.

After the Omnibus

Post-Hoc Tests

A significant omnibus F tells you something limited. At least one group differs from at least one other. It does not say which, and with several groups that is a weak statement. Post hoc tests identify the specific differences. While continuing to control the family wise Type one error rate. Several procedures exist, and they vary in conservativeness. The Bonferroni correction is the most conservative, and the simplest. Use it when a few specific comparisons are planned. Tukey's honestly significant difference is moderately conservative, and widely used. It suits all pairwise comparisons among groups. Scheffe's test is highly conservative, protecting against any possible comparison. And Holm Bonferroni is less conservative than Bonferroni, adjusting alpha sequentially.

A Word About the Name

Post-Hoc Test, Not Post-Hoc Theorizing

One warning about the name post hoc test. It refers to comparisons performed after a significant omnibus F. That is all it refers to. It does not license post hoc theorizing about which comparisons matter. The set of comparisons you examine should be specified by the design. And by the research question. Not selected after seeing which comparisons happen to be significant. Cherry picking among many possible comparisons is a separate kind of error. And no statistical correction can fully repair it. The correction handles the number of tests. It cannot handle a biased selection of them.

Reporting

What a Complete ANOVA Report Contains

A complete ANOVA report contains four things. The F statistic, with both numerator and denominator degrees of freedom. Both of them, because the pair identifies the design. The p value. An effect size estimate, such as eta squared or omega squared. And follow up tests, where appropriate. A typical reporting line reads as follows. F, in parentheses two and twenty seven, equals eight point zero zero. P equals point zero zero one, and eta squared equals point three seven. That last number is a large effect size, by Cohen's conventions. One term is worth carrying forward from all of this. The family wise error rate is the probability of at least one Type one error. Across all the tests conducted in a study. That is what these procedures protect.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Comparing Multiple Groups with ANOVA

Why use ANOVA instead of multiple t tests?
ANOVA tests all group means in one omnibus test, keeping the overall Type I error rate at alpha, while running many t tests inflates it. At alpha = .05, treating the tests as independent, three comparisons give about a .14 chance of at least one false positive (1 - .95 cubed), and ten comparisons give about .40. Three groups have three possible pairs, four have six, and five have ten.
What does the F ratio in ANOVA mean?
The F ratio is the between-groups mean square divided by the within-groups mean square, where a mean square is a sum of squares divided by its degrees of freedom. Within-group variability reflects only noise, while between-group variability reflects noise plus any real effect. If the null hypothesis is true, F is expected to be about 1; real differences between group means push F above 1, and large enough values are significant.
How do you read an ANOVA summary table?
Each row gives a source of variability with its sum of squares (SS), degrees of freedom (df), and mean square (MS). For three groups of 10, between-groups SS = 240 with 2 df gives MS = 120, and within-groups SS = 405 with 27 df gives MS = 15, so F = 120 / 15 = 8.0. The total is 645 with 29 df. The df are k - 1 between, N - k within, and N - 1 total.
Which post hoc test should you use after a significant ANOVA?
The choice depends on which comparisons you need and how conservative you want to be. Tukey's honestly significant difference test is widely used for all pairwise comparisons. Bonferroni suits a few specific planned comparisons, and Holm-Bonferroni is a less conservative sequential version. Scheffé's test is highly conservative and protects against any possible comparison. All of them control the familywise Type I error rate, but none can repair cherry-picking comparisons after seeing the results.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →