Applying Multivariate Experimental Designs, Explained

UPI Study › Courses › Research Methods in Psychology › Free lessons › Lesson 13.4 Free with a free account

Applying Multivariate Experimental Designs

Research Methods in Psychology · Chapter 13: Factorial, Within-Subject, and Small-n DesignsSection 13.4 · lesson 80 of 949 min · free with a free account

← Recognizing Main Effects in Factorial StudiesNext lesson: Working with Within-Subject Designs →

Earn 3 college credits for Research Methods in PsychologyRecommended for credit by NCCRS · Psychology 111 · self-paced · one payment, lifetime access
Enroll in Research Methods in Psychology · $250 →Download the syllabus (PDF)

In this lesson

When one dependent variable cannot capture the effect of interest.

  • Distinguish multivariate from univariate experimental designs
  • Identify questions best answered with several outcomes
  • Explain the role of MANOVA in analyzing them jointly
  • Recognize the common pitfalls in interpretation

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,201 words). Read along, or search it.

Factorial, Within-Subject, and Small-n Designs

Applying Multivariate Experimental Designs

Every design so far has ended with a single measured outcome. This section asks what happens when one outcome is not enough. By the end you will distinguish multivariate from univariate experimental designs. You will identify research questions best answered with multiple dependent variables. You will recognize the role of MANOVA and related techniques in analyzing that data. You will see the advantages of multivariate analysis over multiple univariate tests. Both conceptual advantages and statistical ones. And you will recognize the common pitfalls in interpreting multivariate results.

The Problem

One Number, Many Facets

Many psychological constructs cannot be adequately captured by a single measure. Take anxiety, which is a good example because it is so familiar. Anxiety manifests as physiological arousal. It also manifests as subjective worry, which is a different thing entirely. It shows up as behavioral avoidance. And as cognitive interference, when it disrupts attention and working memory. Reducing all of that to one number ignores most of what anxiety actually is. And anxiety is not unusual in this respect. The same applies to wellbeing, executive function, social skill, and academic performance.

The Missed Pattern

Facets Can Respond Differently

A study that uses one outcome where the construct demands several captures only a slice. And it misses something specific, which is worth naming carefully. It misses any pattern where different facets of the construct respond differently. A treatment might calm the body while leaving the worry entirely untouched. Measure only arousal and you call that a success. Measure only worry and you call the same study a failure. Multivariate experimental designs address that limitation. The independent variables are manipulated exactly as in any other experiment. What changes is that the dependent variables are several rather than one. And the analysis treats those outcomes as a set, rather than one at a time.

Structure

A Vector of Scores, Not a Number

Structurally, a multivariate experiment is not exotic. The elements are the same as in any experiment. Independent variables, random assignment, control of extraneous influences, measured outcomes. Two things change. The number and nature of the outcomes, and the analytic strategy applied to them. Each participant is measured on every dependent variable. So each person produces a vector of scores, rather than a single number. And that changes the question you are asking of the data. It is no longer whether the manipulation affects one variable. It is how the manipulation affects the constellation of variables. Which ones move together, which move in opposition, and which stay unchanged.

The Obvious Alternative

Why Not Just Run Multiple Tests

There is an obvious alternative, and it is worth taking seriously. Just run a separate test on each dependent variable, and summarize the results. That approach has several serious limitations. The first is an inflated Type one error rate, from running many separate tests. A single omnibus test controls the overall error rate instead. The second is that separate tests ignore the correlations among the outcomes. Multivariate analysis uses those correlations. It can detect patterns that are invisible in any single outcome. The third is about effects that are spread thin. An effect can be spread across outcomes while being small in every one of them. The multivariate test is sensitive to combined effects that no individual test would catch.

Two More Reasons

A Joint Summary and a Shared Construct

Two more limitations complete the case. Separate tests provide no joint summary of the effect. You end up with a list of verdicts and no overall statement. Multivariate analysis produces a single coefficient describing the overall impact. And separate tests treat each outcome as theoretically independent. Which, in a well-designed study, they are usually not. Multivariate analysis acknowledges that the outcomes belong to a shared construct. One theme runs under all five of those rows. Separate tests throw away the relationships between the outcomes. And those relationships are exactly where the extra information lives.

A Warning

A Question of Construct, Not Just of Numbers

Now a warning, because this is where designs go wrong at the planning stage. Adding multiple dependent variables is not, by itself, a multivariate design. The real question is whether the variables belong together. Are they facets of a shared construct? Or correlated outcomes of one unified process? When they are, multivariate analysis is appropriate and informative. When they are not, the cleaner choice may be univariate analyses with corrections. Notice what kind of decision that is. It is a theoretical decision, not just a statistical one. No software can tell you whether your outcomes are facets of one thing.

The Standard Tool

Multivariate Analysis of Variance

The most widely used multivariate technique in experimental research has a long name. Multivariate analysis of variance, usually abbreviated to MANOVA. It extends the logic of analysis of variance to multiple dependent variables. What it tests is whether the manipulation produces a difference in the joint mean vector. That is the combined pattern of outcomes, taken across conditions. Now be precise about what a significant multivariate test tells you. It says the conditions differ somewhere in the outcome space. Somewhere is deliberately vague, because the test does not localize the difference. Follow-up univariate tests do that, with appropriate control of error rates. They identify which specific outcomes contribute to the overall difference.

Assumptions

What MANOVA Requires of the Data

Like any statistical technique, MANOVA makes assumptions about your data. It assumes the dependent variables are continuous. It assumes the residuals are approximately normally distributed. And it assumes something about the structure of the relationships between outcomes. Specifically, that the variance and covariance structure is similar across conditions. When those assumptions are violated, you have alternatives. Robust multivariate methods, or permutation-based procedures. The technique itself is widely available in statistical software. And it is the standard tool when a study has several conceptually related outcomes.

Pitfalls One to Three

Where Multivariate Studies Go Wrong

Five pitfalls recur in multivariate work. Take the first three. Including outcomes that are conceptually unrelated to each other. That weakens the multivariate test, and produces results that are hard to interpret. Second, failing to follow up a significant MANOVA with diagnostic analyses. That leaves readers unable to identify which outcomes drove the effect. You have told them something happened, and nothing about what. Third, ignoring redundancy among highly correlated outcomes. Three measures of the same thing are not three facets of a construct. That inflates the apparent dimensionality without adding any information.

Pitfalls Four and Five

Reporting and Sample Size

The remaining two pitfalls concern reporting and sample size. Reporting only the multivariate result, without univariate context. That hides which outcomes were actually sensitive to the manipulation. And using multivariate tests with very small samples. Multivariate analyses require more participants per cell than univariate ones. More participants, for the same adequate power. There is an irony worth noticing in that last one. A multivariate design bought for extra sensitivity can end up less sensitive. Less sensitive than a simple design, if the sample was never large enough to support it.

Two Questions

A Complete Report Answers Both

A well-conducted multivariate analysis answers two questions, in sequence. First, did the manipulation produce any difference in the outcomes considered together? Second, if so, which specific outcomes account for that difference? Reports that answer the first question but not the second are incomplete. They leave readers without the information they actually need. Reports that answer the second without the first are also incomplete. They abandon the protection the multivariate framework was meant to provide. That protection was control of the error rate across many tests. So both layers belong in a complete report. The sequence is the method. Skip either end of it, and the design loses its point.

About Research Methods in Psychology at UPI Study

Research Methods in Psychology (Psychology 111) is a self-paced online course of 94 video lessons in 16 chapters, from the scientific method and research ethics to designs, data collection, sampling, measurement, validity, experiments, descriptive statistics and hypothesis testing. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Compare: Research Methods vs Statistics

Why take it here

Why students take Research Methods in Psychology with UPI Study

How psychologists know what they knowResearch Methods in Psychology teaches how psychological studies are designed, run and judged: research ethics, surveys and observation, correlational and experimental designs, sampling, reliability and validity, and the statistics used to test hypotheses. It is a core course in psychology and many social-science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Research Methods in Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Applying Multivariate Experimental Designs

What is a multivariate experimental design?
A multivariate experimental design manipulates independent variables as usual but measures several dependent variables and analyzes them as a set. Each participant produces a vector of scores instead of a single number. The question becomes how the manipulation affects the whole constellation of outcomes: which move together, which move in opposite directions, and which stay unchanged. Random assignment and control of extraneous variables work just as in other experiments.
Why measure more than one dependent variable in an experiment?
Many psychological constructs have several facets that one measure cannot capture. Anxiety, for example, involves physiological arousal, subjective worry, behavioral avoidance, and cognitive interference. A treatment might calm the body while leaving worry untouched, so measuring only arousal would suggest success and measuring only worry would suggest failure. Measuring several related outcomes reveals patterns in which different facets respond differently.
Why use MANOVA instead of separate tests on each dependent variable?
MANOVA controls the overall Type I error rate with a single omnibus test, while separate tests on each outcome inflate it. It also uses the correlations among outcomes, can detect an effect spread thinly across several measures that no single test would catch, and gives one joint summary of the effect. It fits best when the outcomes are facets of a shared construct; unrelated outcomes are better handled with corrected univariate tests.
What does a significant MANOVA result tell you?
A significant MANOVA tells you the conditions differ somewhere in the combined pattern of outcomes, the joint mean vector, but not which outcomes differ. Follow-up univariate tests with appropriate error-rate control identify the specific dependent variables behind the effect, and a complete report includes both layers. MANOVA assumes continuous outcomes, roughly normal residuals, and similar variance-covariance structure across conditions, and it needs more participants per cell than a univariate analysis.

About the Research Methods in Psychology course

How many college credits is Research Methods in Psychology?
Research Methods in Psychology (Psychology 111) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Research Methods in Psychology cost and what is included?
$250, paid once, with lifetime access: all 94 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Research Methods in Psychology take?
It is self-paced with no deadlines. The 94 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Research Methods in Psychology.

94 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Research Methods in Psychology →Download the syllabusEvery course with PRO →
Research Methods in Psychology
3 credits · NCCRSEnroll →