Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments

By Ben Green, Yiling Chen. 2019

303 citations recorded by the source corpus.

Despite vigorous debates about the technical characteristics of risk assessments being deployed in the U.S. criminal justice system, remarkably little research has studied how these tools affect actual decision-making processes. After all, risk assessments do not make definitive decisions---they inform judges, who are the final arbiters. It is therefore essential that considerations of risk assessments be informed by rigorous studies of how judges actually interpret and use them. This paper takes a first step toward such research on human interactions with risk assessments through a controlled experimental study on Amazon Mechanical Turk. We found several behaviors that call into question the supposed efficacy and fairness of risk assessments: our study participants 1) underperformed the risk assessment even when presented with its predictions, 2) could not effectively evaluate the accuracy of their own or the risk assessment's predictions, and 3) exhibited behaviors fraught with "disparate interactions," whereby the use of risk assessments led to higher risk predictions about black defendants and lower risk predictions about white defendants. These results suggest the need for a new "algorithm-in-the-loop" framework that places machine learning decision-making aids into the sociotechnical context of improving human decisions rather than the technical context of generating the best prediction in the abstract. If risk assessments are to be used at all, they must be grounded in rigorous evaluations of their real-world impacts instead of in their theoretical potential.

Subfields: Political Science and International Relations, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Political Science and International Relations

In Safety Research

In Sociology and Political Science

Citations in the corpus

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “We leave to future work the discussion of decision systems that include discretion beyond the score’s statistical prediction [66].”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “External audit assesses the risk that a system will cause harm to individuals or society from outside the system, and tends to be conducted after deployment, when some harms can already be done (Green & Chen, 2019).”
Discrimination through Optimization (2019, background): “Although most of the work focused exclusively on the algorithms themselves, recently researchers began to point out that auditors should consider the entire socio-technical systems that include the users of those algorithms, an approach referred to as “algorithmin-the-loop” [40, 66].”
The false promise of risk assessments: epistemic reform and the limits of fairness (2020, background): “Two experimental studies found that people are more strongly swayed by a risk assessment’s suggestion to increase estimates of crime risk when evaluating black defendants compared to white defendants [60, 61].”
The ethics of algorithms: key problems and solutions (2020, unknown): “…agreement on the need for algorithmic fairness, particularly to mitigate the risks of direct and indirect discrimination (under US law, ‘disparate treatment’ and ‘disparate impact’, respectively) due to algorithmic decisions (Barocas and Selbst 2016; Grgić-Hlača et al. 2018; Green and Chen 2019).”
Tree Space Prototypes: Another Look at Making Tree Ensembles Interpretable (2016, result): “We note that self-reported user confidence has not been found to be indicative of actual performance and can sometimes even be misleading [15, 27], with studies finding that humans cannot accurately assess their own performance [26, 42].”
The Principles and Limits of Algorithm-in-the-Loop Decision Making (2019, background): “In the context of algorithm-assisted human decision making, prior research has explored topics such as what interactions can facilitate the development of machine learning models [9, 29, 47], how to improve human performance with an algorithm’s assistance [12, 37, 48], and the ways in which laypeopl”
A Framework for Understanding Unintended Consequences of Machine Learning (2019, background): “In some cases, a system that produces results that must first be interpreted by human decision-makers may actually lead to harmful consequences because of phenomena such as automation or confirmation bias (Green and Chen, 2019).”
Consistent Estimators for Learning to Defer to an Expert (2020, methodology): “Machine learning systems are now being deployed in settings to compliment human decision makers such as in healthcare [HAA + 17, RBC + 19], risk assessment [GC19a] and content moderation [LHL16].”
Making Decisions that Reduce Discriminatory Impacts (2019, unknown): “Green & Chen (2019) provide further evidence that when algorithmic risk assessments are shown to human decision-makers the final impact is fraught with unaddressed biases.”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, background): “For example, Green & Chen [16] discovered that self-reported measures could be misleading, since participant’s confidence in their performance was negatively associated with their actual performance.”
This Thing Called Fairness (2019, methodology): “ at the broader process. The criminaljusticesystemhasanimportantdueprocessinterestinprotectingthecalibrationofscores to measured re-arrest risk, so that scores can be interpreted by judges eﬀectively [56]. Arrestees, ontheotherhand,haveastronginterest inminimizingthefalse-positive rateforhigh-riskra”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Cheng et al. [28] compare different explanation interfaces for understanding an algorithm’s university admission decisions.”