Designing Studies with Multiple Groups, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 12.6 Free with a free account

Designing Studies with Multiple Groups

Research Methods in Psychology · Chapter 12: Experimental Design FundamentalsSection 12.6 · lesson 75 of 949 min · free with a free account

← The Logic Behind Matched-Group DesignsNext lesson: Getting Started with Factorial Designs →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

Three or more conditions and the questions they make answerable.

  • Identify questions that need more than two conditions
  • Distinguish dose-response, type-comparison and parametric designs
  • Weigh added conditions against power per cell
  • Apply planned contrasts and post-hoc tests correctly

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,203 words). Read along, or search it.

Experimental Design Fundamentals

Designing Studies with Multiple Groups

Two conditions tell you whether something works. Three or more start telling you how it works. By the end of this section you will identify questions that require more than two conditions. You will design a multi-group experiment with appropriate levels of the independent variable. You will distinguish dose response, type comparison, and parametric designs. You will recognize the trade-off between number of conditions and power per cell. And you will apply planned comparisons and post-hoc tests to interpret the results. That last skill is where most multi-group studies actually succeed or fail.

The Limit of Two

Questions a Two-Group Design Leaves Open

A two-group experiment can establish that a manipulation produces an effect. It then goes quiet on almost every interesting follow-up. Does a higher dose produce a larger effect? And how steep is that relationship? Are some forms of an intervention more effective than others? Does the effect plateau at some point, or decline, or even reverse? Answering any of those requires more than two conditions. And often a careful choice of which conditions to include. Multi-group experiments share the same essential elements as two-group designs. A manipulated variable, random assignment, controlled extraneous variables, a measured outcome. What changes is only the number of levels of the independent variable. And with it, the kinds of inference the design can support.

Structure One

Dose-Response Designs

There are three common multi-group structures. Start with dose response. Dose response designs vary the quantity of a single manipulation across conditions. A drug study might compare placebo, low dose, medium dose, and high dose. A training study might compare different amounts of practice. What this design buys you is the shape of the relationship. Whether it is linear, curvilinear, or threshold based. But reading that shape requires planning the condition spacing carefully. Equal intervals on the dose dimension make trends much easier to interpret. Uneven spacing can make a straight relationship look like a curve.

Structure Two

Type-Comparison Designs

The second structure is the type comparison design. Here the conditions are qualitatively different versions of a manipulation. A study of teaching might compare lecture, discussion, and problem based learning. A memory study might compare visual, auditory, and kinesthetic encoding. Notice that these conditions do not sit on a quantitative dimension. There is no more or less of anything. There are just different kinds. So the question changes shape too. It is which type produces the largest effect, not how much of something is needed. Type comparison designs are common when the practical question is which intervention to deploy.

Structure Three

Parametric Designs

The third structure is the parametric design. It varies one continuous feature of the manipulation across many fine-grained levels. Five, seven, or even ten conditions are common. The goal here is more ambitious than comparison. It is to map the full function relating the independent and dependent variables. Often in order to test the predictions of a quantitative theory. That ambition has a price, and the price is participants. Parametric designs require larger samples, because each cell needs adequate numbers. In exchange, they yield much richer information than coarser designs do.

Resolution vs Power

The Trade-Off You Cannot Escape

Now the trade-off that governs every one of these decisions. Adding conditions improves the resolution of your design. But it reduces the number of participants per cell, which lowers statistical power. Make that concrete with a fixed total sample. Two conditions of fifty participants each, versus four conditions of twenty five each. The four condition version answers more questions. It provides less power per comparison while doing so. So researchers face a real trade-off, not a free upgrade. More conditions answer more questions, but each question is answered less precisely. And here is the practical lesson for your power analysis. Plan power around the comparisons that matter most, not around the omnibus test alone.

Choosing Levels

Spacing Decides What You Can Detect

The choice of levels shapes what the design is capable of detecting. Levels too closely spaced may fail to produce visible differences at all. Levels too widely spaced may step right over an effect in a narrow range. Three principles guide the decision. First, bracket the expected effect. Include levels both below and above where you expect the effect to peak. Second, use theory-relevant spacing. Equal intervals if theory predicts a linear relationship. Denser spacing near the predicted threshold if theory predicts one. Third, anchor with a control. Include a no-manipulation or zero-level condition wherever possible, to establish a baseline.

Two More Principles

Pilot First, Then Plan the Analysis

Two further principles are about timing rather than spacing. Pilot before committing. Run a small pilot to verify that your chosen levels produce detectable differences. Do that before the full study, not as a postmortem afterwards. And plan for the analysis while you are still choosing levels. Choose levels that align with the statistical comparisons you intend to run. Trend analysis needs ordered levels, evenly spaced. Pairwise comparisons suit type comparisons, where order means nothing. Every one of these principles is cheap before data collection. And impossible afterwards.

Analysis Step One

The Omnibus Test

Analysis of a multi-group experiment usually begins with an omnibus test. Most commonly a one-way analysis of variance. Its job is to determine whether any differences exist among the conditions. Now be careful about what a significant omnibus result actually means. It tells you at least one group differs from at least one other group. It does not tell you which groups, or in which direction. With five conditions, that is a very weak statement indeed. So on its own the omnibus test is close to useless for interpretation. Researchers therefore follow it up with more specific comparisons.

Analysis Step Two

Four Kinds of Follow-Up Comparison

There are four common kinds of follow-up comparison. Planned contrasts are comparisons you specified in advance. They are derived from theoretical predictions, and tested with greater statistical power. More power than post-hoc tests, specifically, because you committed to them first. Trend analysis tests for linear, quadratic, or higher-order trends. It requires ordered levels of the independent variable, so it suits dose response designs. Post-hoc pairwise tests examine all possible comparisons among conditions. They carry corrections for multiple testing, which is what costs them power. And control versus treatments comparisons test each condition against one reference. That suits studies where a single control is the natural benchmark.

Before the Data

Why Pre-Specification Pays

One theme runs through all of that, and it is worth stating plainly. Comparisons specified before data collection are statistically more powerful. They are also more credible to readers and to reviewers. Pre-registration formalizes that idea. You list the planned comparisons in a public protocol, before participants are recruited. That is now standard practice in many areas of psychology. And the reason it works is almost embarrassingly simple. Writing down what you expect, before you know the answer, removes the room to drift. It is one of the simplest and most effective safeguards against unintended bias.

Vocabulary

The Terms Worth Carrying Forward

Four terms are worth carrying out of this section. An omnibus test asks whether any differences exist among the conditions. It is a gate, not an answer. A planned contrast is a comparison among specific conditions, specified before data collection. It carries more power than a post-hoc test, because it costs no multiplicity correction. Trend analysis tests for linear or higher-order patterns across ordered levels. Use it when the levels have a natural sequence, and not otherwise. And a post-hoc test is a comparison conducted after the data are examined. Typically with corrections for multiple testing, which is the price of looking first.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Designing Studies with Multiple Groups

Why would an experiment need more than two groups?
More than two groups are needed when the question is how an effect works, not just whether it exists. Extra conditions can show whether a higher dose produces a larger effect, whether some forms of an intervention work better than others, and whether an effect levels off, declines, or reverses. The design keeps the core elements of a two-group experiment; only the number of levels of the independent variable changes.
What is the difference between dose-response, type comparison, and parametric designs?
They differ in how the levels of the independent variable relate to each other. A dose-response design varies the amount of one manipulation, such as placebo, low, medium, and high doses, to reveal the shape of the relationship. A type comparison design tests qualitatively different versions, such as lecture, discussion, and problem-based learning. A parametric design varies one continuous feature across many fine-grained levels, often five to ten, to map the full function.
What is the trade-off of adding more conditions to an experiment?
Adding conditions answers more questions but lowers statistical power for each comparison, because a fixed sample is spread across more cells. For example, 100 participants give 50 per group in a two-group study but only 25 per group across four conditions. For that reason, power analysis should be planned around the comparisons that matter most, not just around the overall omnibus test.
What is the difference between a planned contrast and a post hoc test?
A planned contrast is a specific comparison chosen before data collection from theoretical predictions, while a post hoc test is run after the data are examined, typically across all pairs of conditions. Post hoc tests carry corrections for multiple testing, which costs power, so planned contrasts generally have more power and more credibility. Listing planned comparisons in a pre-registered public protocol makes that commitment verifiable.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →