When data science encodes injustice

A model used to set bail amounts in US courts assigned higher risk scores to defendants from poorer neighbourhoods — and then used the resulting higher incarceration rate to validate itself. This is the circularity at the heart of Weapons of Math Destruction: algorithms trained on biased historical data reproduce and entrench the biases they absorbed.

Cathy O'Neil is a mathematician who worked in hedge funds before becoming a data scientist and then a critic of the industry. Her term 'Weapon of Math Destruction' — a WMD — describes any model that is opaque, operates at scale, and causes damage without accountability. The opacity matters: if you cannot see how a model reaches its conclusions, you cannot challenge an incorrect one.

O'Neil examines a range of systems that meet her definition. Teacher evaluation algorithms in Washington DC fired teachers based on statistical models that later proved unreliable. Predatory online advertising identified financially desperate people and served them loan advertisements with punishing interest rates. College ranking systems warped university priorities by rewarding metrics that could be gamed.

The damage is rarely random. O'Neil argues that WMDs consistently harm the poor and vulnerable, because these are the populations whose behaviour is most thoroughly monitored and whose lack of resources leaves them least able to contest an adverse outcome. Wealthy people, by contrast, are rarely subjected to algorithmic scoring in the same domains.

The book is a call for something like an algorithmic audit process — a requirement that consequential models be made legible and held to standards of accuracy and fairness comparable to those we apply to other professional judgements.