ALGORITHMIC BIAS: WHEN AI DISCRIMINATES AT SCALE

AI increasingly decides who gets arrested, hired, treated, or paroled — and the evidence shows it often decides unequally. Trained on historical data that encodes decades of human bias, algorithmic systems have produced wrongful arrests, discriminatory hiring screens, and unequal medical care, frequently against the people least equipped to contest a machine’s verdict.

Thank you for reading this post, don't forget to subscribe!

When the camera misidentifies

MIT’s Gender Shades project exposed how unevenly facial recognition performs: leading commercial systems from Amazon, IBM, and Microsoft misclassified darker-skinned women far more often than lighter-skinned men, with some missing roughly a third of darker female faces. In the real world, those errors put people in handcuffs — at least eight Americans, nearly all of them Black, have been wrongfully arrested after a face-matching system pointed police at the wrong person. The technology is most error-prone precisely for the groups already over-policed.

Hiring, lending, and healthcare

Bias scales fastest where AI screens people at volume. In 2025, a US federal court let a collective action proceed alleging Workday’s AI screening disadvantaged applicants over 40 — echoing Amazon’s earlier, scrapped recruiting tool that penalized résumés containing the word “women’s.” In healthcare, a widely used risk algorithm studied in Science systematically under-referred Black patients for extra care; correcting the bias would have more than doubled the share of Black patients flagged for help. The pattern: the model looks neutral, but the training data is not.

“The model looks neutral, but the training data is not — bias scales fastest where AI screens people at volume.”

Justice by algorithm

Risk-scoring tools now inform bail, sentencing, and parole. ProPublica’s investigation of COMPAS found it labeled Black defendants who did not reoffend as “high risk” at nearly twice the rate of comparable white defendants, while white defendants who did reoffend were more often rated low risk. Because such scores carry an air of objectivity, they can entrench discrimination while making it harder to challenge.

The governance response

Regulators are beginning to treat fairness as a requirement, not an aspiration. South Korea’s AI Framework Act (effective 2026) mandates non-discrimination for high-impact AI in areas like healthcare and public services, the EU AI Act classes such uses as “high-risk,” and US agencies and courts are testing existing civil-rights law against algorithmic decisions. The throughline: when AI makes a high-stakes call, someone must remain accountable for auditing it and for the people it gets wrong.