Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

By Shikha Verma. 2016

954 citations recorded by the source corpus.

A former Wall Street quant sounds an alarm on the mathematical models that pervade modern life and threaten to rip apart our social fabricWe live in the age of the algorithm. Increasingly, the decisions that affect our liveswhere we go to school, whether we get a car loan, how much we pay for health insuranceare being made not by humans, but by mathematical models. In theory, this should lead to greater fairness: Everyone is judged according to the same rules, and bias is eliminated. But as Cathy ONeil reveals in this urgent and necessary book, the opposite is true. The models being used today are opaque, unregulated, and uncontestable, even when theyre wrong. Most troubling, they reinforce discrimination: If a poor student cant get a loan because a lending model deems him too risky (by virtue of his zip code), hes then cut off from the kind of education that could pull him out of poverty, and a vicious spiral ensues. Models are propping up the lucky and punishing the downtrodden, creating a toxic cocktail for democracy. Welcome to the dark side of Big Data. Tracing the arc of a persons life, ONeil exposes the black box models that shape our future, both as individuals and as a society. These weapons of math destruction score teachers and students, sort rsums, grant (or deny) loans, evaluate workers, target voters, set parole, and monitor our health. ONeil calls on modelers to take more responsibility for their algorithms and on policy makers to regulate their use. But in the end, its up to us to become more savvy about the models that govern our lives. This important book empowers us to ask the tough questions, uncover the truth, and demand change.

Subfields: Sociology and Political Science, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

In Social Sciences & Humanities

Citations in the corpus

Measurement and Fairness (2019, background): “and potential consequences of reliance on EVAAS [3]. Teachers lost their jobs and careers due to this score, which was also used to distribute resources. This led to schools manipulating their scores [52]; in addition, setting such objectives leads educators and schools to \teach to the test," ”
A Survey on Bias and Fairness in Machine Learning (2019, background): “However, like people, algorithms are vulnerable to biases that render their decisions “unfair” [5, 100].”
Large image datasets: A pyrrhic win for computer vision? (2020, background): “Image classification and labelling practices, without the necessary precautions and awareness of these problematic histories, pick up these stereotypes and prejudices and perpetuate them [35, 73, 74].”
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence (2020, background): “While these recent innovations have led to some societal benefits, they have also demonstrated their potential to be abused or misused in ways their designers could not have imagined (O’Neil, 2016).”
Fairness in Machine Learning: A Survey (2020, background): “[56] articulate the challenge of sampling for fairness as the following question: how are samples selected from a (large) dataset that are both diverse in features and fair to sensitive attributes? Without care, sampling can propagate biases within the training data [213, 88], as ensuring diversity ”
The Disparate Effects of Strategic Manipulation (2018, background): “Theoretically, algorithms have great potential to reform existing systems to become both more efficient and equitable, but as exposed by various high-profile investigations [1, 2, 3, 4], prediction-based models that make or assist with consequential decisions are, in practice, highly prone to reprod”
The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning (2018, background): “As the influence and scope of these risk assessments increase, academics, policymakers, and journalists have raised concerns that the statistical models from which they are derived might inadvertently encode human biases (Angwin et al., 2016; O’Neil, 2016).”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “d even criminal justice [21]. However, these algorithms that serve critical roles in many industries have their own disadvantages that can result in discrimination [24,118] and unfair decision-making [92]. For instance, recently, news feed and targeted advertising algorithms in social media have tak”
AI Ethics in Predictive Policing: From Models of Threat to an Ethics of Care (2019, background): “There is a growing body of research examining the ways in which data-driven algorithms are being used in an increasing number of critical decision processes, often with little or no accountability [6]–[9], and sometimes with little or no real understanding of how they function in the real world or w”
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse (2019, background): “In the years since, popular texts have probed the negative impacts of big data and algorithmic decision-making in a range of contexts (e.g., O ’ Neil, 2016).”
Identifying Significant Predictive Bias in Classifiers (2016, background): “the original training data, disparate impacts of predictions, and adjusting predictions to ensure fairness [1, 4, 6, 15] — in areas like criminal justice [2, 3, 7, 17], but also other various sectors [9, 11, 14, 19]. We focus here on predictive bias. Existing literature on predictive bias has focuse”
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention (2021, background): “Some scholars and opinion makers are even warning that a continuous adoption of these technologies in our daily lives is bound to increase inequality and harm disadvantaged groups (O’Neil, 2016).”