UPI Study › Courses › Abnormal Psychology › Free lessons › Lesson 2.3 Free with a free account
Boundaries and Shortcomings of Clinical Studies
Abnormal Psychology · Chapter 2: Investigating Mental Health: Research FoundationsSection 2.3 · lesson 8 of 9717 min · free with a free account
Free preview
Create a free account to keep watching
It takes a minute and costs nothing. Your free account unlocks all of Chapter 1 and 3 more lessons of Abnormal Psychology.
Lesson complete
Earn the 3 college credits for Abnormal Psychology
Enrolling adds a quiz after each lesson, the graded work and the official transcript. Recommended for credit by NCCRS.
Next lesson: A Primer on Biological Psychology →← The Purpose and Function of Control GroupsNext lesson: A Primer on Biological Psychology →
In this lesson
What bounds every clinical finding, and how to read one calibrated rather than cynical.
- Identify the ethical constraints, including consent where capacity is affected
- Describe the sampling and measurement limits that bound any finding
- Explain publication bias, replication difficulty, and the reforms adopted
- Separate statistical significance from effect size
Transcript
Everything said in the video, section by section (2,336 words). Read along, or search it.
Investigating Mental Health: Research Foundations
Boundaries and Shortcomings of Clinical Studies
This section is about what clinical research cannot do, and why. Consent, when the condition under study affects the ability to give it. Questions that cannot ethically be settled at all, permanently. Samples that narrow so sharply they no longer resemble a caseload. Measurement that can never be more precise than the diagnosis behind it. Participants who leave, in ways that manufacture results from nothing. And a literature shaped by what gets published rather than what was found. Then the single most misread pair of words in research. Significant, and large. None of that makes the research worthless. It makes reading it a skill, and this section is that skill.
Before You Begin
Calibrated, Not Cynical
Two notes before we begin. This material is educational, and not a basis for diagnosing anyone. And every study described here is a composite constructed for instruction. No real trial, and no real researcher's work, is criticized anywhere. Now a framing note, and it matters more than usual here. A section on the shortcomings of research can read as saying the research is worthless. It is not saying that. Every limitation here was identified by researchers themselves. Each is discussed openly in the literature, and each has prompted concrete reform. The aim is to make you a calibrated reader. Someone who can say what a study does and does not support. Rather than a cynical one, who has swapped uncritical belief for uncritical dismissal.
Consent
When the Condition Under Study Affects Capacity
Informed consent is a participant's voluntary agreement to take part. Given after they understand what the research involves, and what risks it carries. What alternatives exist, and that they may withdraw at any time. Withdraw without any effect at all on their care. That last component is the one most often weakened in practice. A participant who believes their treatment depends on continuing has not consented voluntarily. Capacity to consent is the ability to understand the information, and retain it. To weigh it, and to communicate a choice. Four abilities. Several of the conditions most in need of study can affect those abilities. Someone in severe depression may understand it perfectly, and decide from hopelessness. Capacity is decision-specific and fluctuating, and it is not a property of a diagnosis. Where capacity is absent or partial, two mechanisms apply. Proxy consent is permission given by a legally authorized representative. Usually restricted to low-risk research, or research offering a prospect of direct benefit. Assent is the person's own agreement, sought wherever they can express a view. It exists because someone who cannot give legal consent can still meaningfully refuse. And excluding people who cannot consent is not a neutral solution. The conditions in greatest need of evidence become the ones least studied.
Review
Weighing Risk Against Knowledge
Before a study begins, an independent research ethics committee examines it. Researchers, clinicians, and lay members, independent of the research team. The central task is risk-benefit review. Whether the risks to participants are justified by the knowledge the study would produce. First, it is not a calculation. Risks and benefits are not measured in the same units. And they usually fall on different people. Participants bear the risk, while future patients receive the benefit. Second, it attends to vulnerability. Inpatients. People dependent on the researcher for their care. Prisoners. And people in financial need, for whom payment becomes pressure. Third, it applies scientific standards as an ethical matter, which surprises people. A study too small to answer its question exposes participants to risk for nothing. That is an ethical failure, and not merely a methodological one. Confidentiality also carries specific difficulties in clinical samples. The data are sensitive, and the samples are often small. So identifying information is separated from research data, and access is restricted. And the limits of confidentiality are stated at consent, because no researcher can promise absolute secrecy.
The Undoable Study
Some Questions Cannot Be Settled, Permanently
Here is the recurring frustration of clinical research. The design capable of answering a question is frequently the design that must not be used. Does prolonged childhood neglect cause psychological difficulty in adulthood? The design that would settle it is an experiment. Take a large group of children. Assign them by chance to neglect, or to ordinary care. Then follow both groups for twenty years. Random assignment would eliminate every third variable that makes observational evidence ambiguous. The study is monstrous, and no ethics committee would consider it for a moment. The important observation is not that it is forbidden. It is that being forbidden has a permanent consequence for what can be known. That causal question cannot be settled to the standard an experiment would provide. So researchers assemble converging evidence of weaker kinds. Prospective longitudinal studies, following children already exposed. Natural experiments, where circumstances outside anyone's control separated similar children. Sibling comparisons, which hold family background constant. And animal work, whose applicability to humans is itself contestable. Together, when they agree, they support a conclusion held with reasonable confidence. Reasonable, but not experimental. That is the honest position.
Who Is In It
The Distance Between the Sample and the Caseload
Every finding describes the people who were actually in the study. The question is how much distance lies between them and the people a clinician sees. A clinical sample is drawn from people already receiving treatment. A community sample is drawn from the general population, whether or not they sought help. People in clinical samples have recognized a problem, obtained a referral, and been able to attend. So they differ in help-seeking, in insight, in resources, and in access. A condition studied only in clinical samples looks more severe than it is. Now picture a funnel, using hypothetical figures constructed for instruction. Ten thousand people in the population at the top. Eligibility criteria are applied. Then voluntary consent is sought. Two hundred and forty remain in the analyzed sample. Two point four percent. Every stage represents a defensible decision, and consent is a requirement rather than a failing. But the finding is a statement about the two hundred and forty. Extending it to the rest is an inference the study itself does not support. Exclusion criteria narrow the sample further, and they add up. Additional diagnoses, substance use, elevated risk, other medication. Each exclusion is individually defensible. Cumulatively they remove most of a real caseload. The comorbidity exclusion is the sharpest, because carrying more than one diagnosis is typical.
How Far It Travels
Generality Is Assumed, Not Demonstrated
Generalizability is the extent to which a finding holds beyond the people studied. The evidence base in psychology is drawn from a narrow slice of the world. Participants in wealthy, industrialized, English-speaking countries are heavily overrepresented. And within those countries, disproportionately from university populations. It does not mean findings from such samples are false. It means their generality is assumed rather than demonstrated. And where it has been tested, the results have varied by finding. Some replicate widely across populations, and others do not. Culture shapes how difficulty is expressed and reported. Language affects instruments directly. A rating scale that is translated but not revalidated may not measure the same thing. Age matters because instruments developed with working-age adults get applied to adolescents. And to older adults, with less evidence than the usage implies. And comorbidity matters, because the evidence base was largely built by excluding it.
Measurement
A Finding Cannot Be More Precise Than Its Category
Clinical research measures things that cannot be observed directly. A symptom scale is a proxy measure. An observable stand-in for a construct that cannot itself be measured. A questionnaire score is not anxiety. It is a set of answers about anxiety. The two are related without being the same thing. Self-report accuracy carries social desirability, recall bias, and the limits of insight. Scales are often not equivalent to one another. So two trials reporting improvement may have measured different things entirely. And a change of a given number of points does not mean the same across instruments. Rating scales are also more sensitive to change than to state. They detect movement well, and are poor at deciding whether a threshold has been crossed. Above all of these sits a ceiling, and it comes from an earlier chapter. Any study recruiting participants by diagnosis inherits diagnostic reliability. If clinicians do not agree well about who has a condition, the sample is imprecise. And no amount of care in the rest of the design can repair that. A finding about a category cannot be more precise than the category.
Attrition
Who Leaves Matters More Than How Many
Attrition is the loss of participants before a study is complete. The instinct is to treat it as a question of quantity. A study losing ten percent is better than one losing forty percent. That is true, and it is the less important half of the problem. What matters more is whether those who left differ from those who stayed. Suppose participants who were not improving drop out of the treatment arm. While those doing well remain in it. The final analysis then compares the successes of one group. And the treatment appears effective for that reason alone. This is differential attrition. Dropout that differs between arms, or relates to the outcome. It can manufacture an entire result from nothing. So thirty percent attrition falling equally across both arms is a smaller threat. Smaller than ten percent concentrated among non-responders in one arm. The main protection is intention-to-treat analysis. Every participant is analyzed in the group they were assigned to. It systematically underestimates how well a treatment works for those who take it. And it is used anyway, for a good reason. That discards the one feature which made the comparison worth anything.
The File Drawer
Publication Bias, Replication, and the Reforms
Publication bias is the tendency for positive or striking results to be published. While studies finding no effect are not. It arises from ordinary incentives rather than misconduct. And researchers, knowing this, set an unpublished null result aside. That last step gives the phenomenon its other name. The file-drawer problem. Suppose ten teams test a treatment that does nothing at all. Chance alone will produce an apparently positive result in some of them. If those are published and the null studies are not, the literature shows a treatment that works. The distortion is invisible from inside, because the missing studies are missing. Selective outcome reporting publishes the outcomes that reached significance. From among the several that were measured, without saying the others were collected. And analytic flexibility allows a null result to be worked into a positive one. Now replication. From the early twenty tens, systematic attempts repeated published findings. They recovered the original result far less often than researchers had expected. And where effects were recovered, they were frequently smaller than originally reported. A failed replication does not establish that the original finding was false. But a body of failed replications does show the original confidence was too high. The responses are concrete. Pre-registration deposits the plan publicly before data collection. The hypothesis, the design, the primary outcome, and the analysis plan. Registered reports go further. The journal commits to publishing before the data exist. That removes the incentive entirely, because acceptance no longer depends on the outcome. These reforms are real, and their coverage is partial. That is the accurate summary.
Two Different Questions
Significance Is Not Size
Two terms are routinely confused, and the confusion produces more misinterpretation than any other. Statistical significance is a statement about how likely a result of this size would be. If there were really no effect at all. It does not mean the effect is large, important, or clinically useful. The everyday sense of significant, meaning substantial, is exactly the wrong reading. Effect size is a statement about how big the difference actually is. It answers the question a clinician has. How much better will this person be? The two come apart in both directions, and the reason is sample size. Take two hypothetical findings, constructed for instruction. Finding A has twenty thousand participants. The treatment reduces the average score by one point two points, on a scale of one hundred. That is highly significant, because with that many participants almost anything is. And the effect is very small. Most participants would notice no difference in daily life. Finding B has thirty-eight participants, and a reduction of nineteen points. That is not significant, because thirty-eight participants cannot rule out chance. But the effect is large, and imprecisely estimated. The impressive significance sits with the trivial effect. The substantial effect has no significance at all. Finding A should not be adopted on significance alone. Finding B should be replicated in a larger, adequately powered sample. And a null result from an underpowered study is an absence of evidence. Which is a different thing from evidence of absence.
Worked Through
A Headline Trial, and a Calibrated Conclusion
Now put the whole checklist onto one trial, which is a composite describing no real study. Two hundred and forty adults were assigned by chance to an eight-week program, or a waitlist. Anyone with an additional diagnosis or substance use was excluded. So was anyone at elevated risk, or without fluent command of the language of assessment. Two hundred and forty entered. One hundred and sixty-eight completed. The primary outcome was a self-report anxiety scale at eight weeks. The treated group improved by eleven points, against four in the control arm. Reported as statistically significant, and analyzed by completers only. Start with design. It is randomized with a control group, which is genuinely good. That handles regression, spontaneous remission, natural course, and repeated measurement. Expectancy is not handled. Participants knew which arm they were in. Now the sample. The exclusions remove most of a real caseload. The comorbidity exclusion alone removes the typical patient. Now attrition, which is the most serious problem here. Seventy-two of two hundred and forty left, and fifty-two of those from one arm. Completer-only analysis then compares those who stayed in the treatment. If people left because they were not improving, this result appears with no treatment effect. And on magnitude, significance is reported while no effect size is given. That omission is itself informative. So the calibrated conclusion. In a narrowly selected sample, among those who completed it. The program was associated with greater self-reported improvement than a waitlist. Whether that reflects the content, or expectancy, or differential dropout, cannot be determined. Notice that this conclusion is neither acceptance nor dismissal.
About Abnormal Psychology at UPI Study
Abnormal Psychology (Psychology 180) is a self-paced online course of 97 video lessons in 25 chapters, from the main theoretical perspectives and clinical assessment to anxiety, mood, psychotic and personality disorders, treatment and the law. It carries a NCCRS credit recommendation for 3 semester credits.
Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →
Why take it here
Why students take Abnormal Psychology with UPI Study
Where the credit goes
Universities that work with NCCRS
Abnormal Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Thomas Edison State UniversitySigned partner
University of the PeopleSigned partner
Charter Oak State CollegeSigned partner
Strayer UniversitySigned partner
Columbia Southern UniversitySigned partner
Indiana TechSigned partner
Penn StateWorks with NCCRS
Southern New Hampshire UniversityWorks with NCCRS
SUNYWorks with NCCRS
+many more cooperating universitiesFind my college →Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.
FAQ
Questions students ask us
About Boundaries and Shortcomings of Clinical Studies
What is the difference between statistical significance and effect size?
What is publication bias, or the file-drawer problem?
Why is intention-to-treat analysis used in clinical trials?
How do researchers get informed consent when a mental illness affects decision-making capacity?
About the Abnormal Psychology course
How many college credits is Abnormal Psychology?
How much does Abnormal Psychology cost and what is included?
Which universities accept UPI Study credits?
How long does Abnormal Psychology take?
How are the exams done?
How do my credits reach my college?
Ready for the full course? Earn 3 credits in Abnormal Psychology.
97 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.
3 credits · NCCRSEnroll →