The age of secrecy and unfairness in recidivism prediction

By Cynthia Rudin, Caroline Linjun Wang, Beau Coker. 2018

200 citations recorded by the source corpus.

In our current society, secret algorithms make important decisions about individuals. There has been substantial discussion about whether these algorithms are unfair to groups of individuals. While noble, this pursuit is complex and ultimately stagnating because there is no clear definition of fairness and competing definitions are largely incompatible. We argue that the focus on the question of fairness is misplaced, as these algorithms fail to meet a more important and yet readily obtainable goal: transparency. As a result, creators of secret algorithms can provide incomplete or misleading descriptions about how their models work, and various other kinds of errors can easily go unnoticed. By partially reverse engineering the COMPAS algorithm -- a recidivism-risk scoring algorithm used throughout the criminal justice system -- we show that it does not seem to depend linearly on the defendant's age, despite statements to the contrary by the algorithm's creator. Furthermore, by subtracting from COMPAS its (hypothesized) nonlinear age component, we show that COMPAS does not necessarily depend on race, contradicting ProPublica's analysis, which assumed linearity in age. In other words, faulty assumptions about a proprietary algorithm lead to faulty conclusions that go unchecked without careful reverse engineering. Were the algorithm transparent in the first place, this would likely not have occurred. The most important result in this work is that we find that there are many defendants with low risk score but long criminal histories, suggesting that data inconsistencies occur frequently in criminal justice databases. We argue that transparency satisfies a different notion of procedural fairness by providing both the defendants and the public with the opportunity to scrutinize the methodology and calculations behind risk scores for recidivism.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…issues with explanations have arisen with saliency maps (Adebayo et al., 2018; Chen et al., 2019), assessment of fairness and variable importance (Rudin et al., 2020; Dimanov et al., 2020), and uncertainty bands for variable importance (Gosiewska and Biecek, 2020; Fisher et al., 2019), as well…”
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2018, background): “In fact, COMPAS seems to be nonlinear, and it is entirely possible that COMPAS does not depend on race (beyond its correlations with age and criminal history) [27].”
Please Stop Explaining Black Box Models for High Stakes Decisions (2018, methodology): “As a important side note, typographical errors seem to be common in computing COMPAS, which sometimes determine bail decision outcomes [Wexler, 2017, Rudin et al., 2018].”
Characterizing Fairness Over the Set of Good Models Under Selective Labels (2021, methodology): “COMPAS is a proprietary risk assessment developed by Northpointe (now Equivant) using up to 137 features (Rudin et al., 2020).”
In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making Processes (2019, background): “However, we recognize that some models are kept secret over concerns of being proprietary or to maintain a competitive advantage [37].”
A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning (2019, background): “[37] Cynthia Rudin, Caroline Wang, and Beau Coker.”
Measuring Bias (2023, methodology): “We use race as the protected group, but could also be age [20], [31].”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results (2018, methodology): “…for specific variable selection procedures (forward stepwise regression, least angle regression, and the lasso regularization path), and Berk et al. (2013) present a framework that holds for all variable selection procedures that is more conservative than Scheffé protection (Scheffé 1959).”
All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance (2018, result): “ of priors, felony indication, or their unmeasured proxies by a factor lower than 1.77, then intervals as high as what we observe would occur with probability <0:05. This result is consistent with Rudin et al. (2019), who nd age to be highly predictive of COMPAS score. It is worth noting that the”
Limits and Possibilities for “Ethical AI” in Open Source: A Study of Deepfakes (2022, background): “unfairness in classification or prediction systems, arguably harder to accomplish when the model and data is proprietary [14, 66].”
When Combating Hype, Proceed with Caution (2021, background): “…of fairness on some measures would be satisfactory: Fairness norms can conﬂict with one another, and in some cases, a machine decision-maker will be given more trust and deference than a human decision-maker would in the same situation (see, e.g., Rudin et al., 2020; Fazelpour and Lipton, 2020).”
The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis (2019, background): “ast one serious aw: COMPAS seems to depend non-linearly on age, and once that nonlinearity is considered and subtracted out, machine learning algorithms provide no evidence to back ProPublica’s claim [47]. This shows one of the dangers of trying to explain black box models { the variables that are i”
The explanation game: a formal framework for interpretable machine learning (2020, unknown): “Subsequent studies using the same dataset reached different conclusions regarding the algorithm’s reliance on race (Fisher et al. 2019; Rudin et al. 2018), high- lighting the inherent uncertainty of model reconstruction when the target algorithm is not forthcoming.”