Fair for Everyone: Tackling Bias in Assessments, Explained

UPI Study › Courses › Educational Psychology › Free lessons › Lesson 8.11 Free with a free account

Fair for Everyone: Tackling Bias in Assessments

Educational Psychology · Chapter 8: Assessment, Testing, and Making Sense of ResultsSection 8.11 · lesson 80 of 1009 min · free with a free account

← When Tests Have Too Much Power: High-Stakes TestingNext lesson: Using Tests Responsibly: Ethics in Assessment →

Earn 3 college credits for Educational PsychologyRecommended for credit by NCCRS · Psychology 120 · self-paced · one payment, lifetime access
Enroll in Educational Psychology · $250 →Download the syllabus (PDF)

In this lesson

Where bias enters an assessment, how it is detected, and what fairness requires.

  • Define test bias and distinguish its major forms
  • Explain differential item functioning analysis
  • Describe the role of bias review committees
  • Practice culturally responsive assessment design

Transcript

Everything said in the video, section by section (1,159 words). Read along, or search it.

Assessment, Testing, and Making Sense of Results

Fair for Everyone

A group of students scoring lower on a test is not, by itself, bias. Bias is a claim about where that difference comes from. This section is about making that claim precisely, and testing it. We start with the definition, which is narrower than most people assume. Then four sources of bias: cultural, linguistic, format, and income. Then how bias is detected statistically, through DIF analysis. Then bias review committees, which catch what statistics cannot. And finally culturally responsive assessment, which goes further. One thing to keep in mind through all of it. Bias is almost never the product of anyone's bad intent. It is the product of blind spots, which is why systems are needed.

Definition

Bias Is a Property of the Test

Here is the definition, and every word of it is doing work. Test bias occurs when an assessment systematically misestimates ability. Under- or overestimates it, for a particular group of test-takers. And it does so due to characteristics of the test. Rather than characteristics of the examinees. That final clause is the entire distinction. A score gap between groups is a finding, and not yet a diagnosis. The bias claim says the gap originates in the instrument itself. There is a technical term for what bias introduces. Construct-irrelevant variance. Score differences caused by factors unrelated to what is being measured.

Cultural

Assuming a Background the Student Does Not Have

Cultural bias is the most discussed source. It arises when items assume background knowledge or experiences. Knowledge more common in dominant cultural groups. The source gives three concrete examples. References to sailing. References to golf. Or references to particular foods. Any of those may be unfamiliar to students from different backgrounds. And that unfamiliarity creates an irrelevant source of difficulty. Irrelevant, because it has nothing to do with the construct measured. Notice what may be perfectly fine about such an item. The mathematics inside it can be completely fair. It is the scenario wrapped around the mathematics that is not.

Linguistic

Language Difficulty the Construct Does Not Require

Linguistic bias works through a similar mechanism. It occurs when the vocabulary or sentence complexity of items exceeds the language proficiency of English language learners. And here is the crucial condition attached to that. It happens even when the content being tested does not require advanced English. Mathematical reasoning, for instance. So one test has quietly become two tests. The student must pass a reading test before reaching the math test. And only the second of those was intended. The mitigation is straightforward in principle. Plain language guidelines, and accommodations for language learners.

Format And Income

Two Biases That Are Not About Content

Two more sources of bias are not about item content at all. The first is gender format bias. Research shows girls tend to score relatively higher on constructed-response items. And relatively lower on multiple-choice items, compared to boys. And that pattern holds independent of actual knowledge differences. Which proves something important about assessment design. The format of an assessment can introduce differential effects. Not just its content. The container matters too. The second is socioeconomic bias, which intersects all the others. Students from lower-income families may have practiced test formats less. They may face higher levels of test anxiety. And they may lack access to test preparation resources their peers have.

DIF

Detecting Bias Statistically

So how is bias actually detected in a real test? One method is differential item functioning analysis. It is a statistical technique for identifying suspicious items. Items that perform differently for two groups of equal ability. Here is the example the source gives. A math item is answered correctly by seventy percent of White students. But by only forty-five percent of Black students. Students who have the same total math score. That item may contain cultural or linguistic bias. And it should be examined carefully. The phrase equal ability is what makes the flag meaningful. The comparison is between students the test already scored the same.

Its Limits

What Statistics Cannot Catch

DIF analysis is not an unusual or optional step. It is a standard part of test development for major assessments. But it has a ceiling, and the ceiling matters. It catches only statistically detectable bias. It cannot eliminate subtler forms of cultural bias. Bias embedded in item content, and in item context. Think about why that limitation exists. A statistical flag needs a large enough effect to reach significance. Below that threshold, the item looks entirely clean. So a statistical screen alone is not a fairness program. Something has to read the items with human eyes.

Bias Review

A Panel That Reads Every Item

That something is the bias review committee. These are panels of educators from diverse backgrounds. Cultural, linguistic and demographic diversity is the point of the panel. They review test items during the development process. Not afterward, which would be too late to matter. They look for three specific things. Language that may be unfamiliar or offensive to specific groups. Contexts that disadvantage particular cultural groups. And stereotypes that might trigger stereotype threat. Items flagged by the committee are revised, or eliminated. No process can make an assessment completely bias-free. But systematic bias review substantially improves fairness.

Six Sources

Mechanism, and the Matching Mitigation

Six sources of bias, each with a mechanism and a mitigation. Cultural bias works by assuming dominant cultural knowledge. Mitigate it with bias review committees, and diverse item writers. Linguistic bias adds language difficulty irrelevant to the construct. Mitigate it with plain language guidelines and accommodations. Gender format bias means a format advantages one gender. Mitigate it by using mixed formats, and analyzing results by gender. Socioeconomic bias runs on test preparation access inequity. Mitigate it with free prep resources, and time accommodations. Stereotype threat works through anxiety about group stereotypes. Mitigate it with an affirming environment and identity-safety cues. And DIF items show statistical group performance differences. Those are caught by DIF analysis during test development.

Beyond Reduction

Culturally Responsive Assessment

Everything so far has been about reducing harm. Culturally responsive assessment goes further than that. It actively affirms what students bring with them. Their cultural knowledge, their experiences, and their strengths. In practice, that means three things. Designing tasks that use culturally familiar contexts. So students demonstrate knowledge through settings they recognize. Offering multiple modes of response. Oral, written, visual, or performative, to fit different communication styles. And engaging families and communities in understanding the data. Teachers who practice this report two results, not one. Higher engagement, and more accurate pictures of student learning.

In Practice

Idioms, and Half a Gap

Here is bias detection and repair in a single district. A literacy coordinator studies the reading gap in her data. The gap between English language learners and native speakers. She notices it is not evenly spread across the test. It is largest on items with complex figurative language. Idioms and metaphors specific to American cultural contexts. So she works with the test development team to revise those items. Replacing that language with more universally comprehensible expressions. But only where doing so does not change the reading skill measured. That condition is what keeps this from being a lowered standard. The revision reduces the gap on those items by nearly half. Nobody in that story intended any harm at any point. The items came from within one cultural frame of reference.

About Educational Psychology at UPI Study

Educational Psychology (Psychology 120) is a self-paced online course of 100 video lessons in 11 chapters, from memory, conditioning and child development to motivation, individual differences, assessment, instruction and educational technology. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Psychology and social science courses Every course with PRO →

Why take it here

Why students take Educational Psychology with UPI Study

How people learn, and how to teach themEducational Psychology explains how students learn, develop and stay motivated, and how teachers assess and support them. It is a standard course in education and teaching degrees and a common psychology elective.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Educational Psychology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Fair for Everyone: Tackling Bias in Assessments

What is test bias?
Test bias occurs when an assessment systematically underestimates or overestimates the ability of a particular group because of characteristics of the test rather than of the test-takers. A score gap between groups is not, by itself, proof of bias. Bias introduces construct-irrelevant variance, meaning score differences caused by factors unrelated to the skill being measured, such as unfamiliar cultural references.
What is differential item functioning (DIF) analysis?
Differential item functioning analysis is a statistical technique that flags test items performing differently for two groups of equal ability. For example, if 70 percent of one group and 45 percent of another answer an item correctly despite having the same total math score, the item may contain cultural or linguistic bias. DIF is standard in major test development but misses subtler bias in item content.
What does a bias review committee do?
A bias review committee is a panel of educators from diverse cultural, linguistic and demographic backgrounds who read test items during development to catch bias that statistics miss. They look for unfamiliar or offensive language, contexts that disadvantage particular cultural groups, and stereotypes that could trigger stereotype threat. Flagged items are revised or removed, which substantially improves fairness, though no process makes a test completely bias-free.
What is linguistic bias in testing?
Linguistic bias occurs when a test's vocabulary or sentence complexity exceeds an English language learner's proficiency even though the skill being tested, such as mathematical reasoning, does not require advanced English. The student effectively has to pass a reading test before reaching the math test. Plain language guidelines and accommodations for language learners help reduce this source of bias.

About the Educational Psychology course

How many college credits is Educational Psychology?
Educational Psychology (Psychology 120) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Educational Psychology cost and what is included?
$250, paid once, with lifetime access: all 100 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Educational Psychology take?
It is self-paced with no deadlines. The 100 lessons total about 15 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Educational Psychology.

100 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Educational Psychology →Download the syllabusEvery course with PRO →
Educational Psychology
3 credits · NCCRSEnroll →