The Principles and Limits of Algorithm-in-the-Loop Decision Making

By Ben Green, Yiling Chen. 2019

424 citations recorded by the source corpus.

The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an "algorithm-in-the-loop'' process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decisions. Society lacks both clear normative principles regarding how people should collaborate with algorithms as well as robust empirical evidence about how people do collaborate with algorithms. Given research suggesting that people struggle to interpret machine learning models and to incorporate them into their decisions---sometimes leading these models to produce unexpected outcomes---it is essential to consider how different ways of presenting models and structuring human-algorithm interactions affect the quality and type of decisions made. This paper contributes to such research in two ways. First, we posited three principles as essential to ethical and responsible algorithm-in-the-loop decision making. Second, through a controlled experimental study on Amazon Mechanical Turk, we evaluated whether people satisfy these principles when making predictions with the aid of a risk assessment. We studied human predictions in two contexts (pretrial release and financial lending) and under several conditions for risk assessment presentation and structure. Although these conditions did influence participant behaviors and in some cases improved performance, only one desideratum was consistently satisfied. Under all conditions, our study participants 1) were unable to effectively evaluate the accuracy of their own or the risk assessment's predictions, 2) did not calibrate their reliance on the risk assessment based on the risk assessment's performance, and 3) exhibited bias in their interactions with the risk assessment. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

Subfields: Artificial Intelligence, Safety Research, AI Ethics, Fairness & Society, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In AI Ethics, Fairness & Society

In Human-Computer Interaction

Citations in the corpus

Measurement and Fairness (2019, background): “ribute to the meaning of the test scores" [44]. Indeed, interacting with measurements can create room for more individual judgments, automation bias, and (lack of) intuition and interpretability [6, 16, 28, 38, 55, 60]. That is, as measurements themselves may change the world in possibly uninte”
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “72,73 Providing uncertainty estimates along with explanations may also improve trust calibration, but research remains to be done in this area.”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Consistent Estimators for Learning to Defer to an Expert (2020, methodology): “These models are either used as a tool to help the downstream human decision maker with judges relying on algorithmic risk assessment tools [GC19b] and risk scores being used in the ICU [FHH + 17], or instead these learning models are solely used to make the ﬁnal prediction on a selected subset of…”
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems (2020, methodology): “Although not universally effective [38], these strategies have shown promising results as they improve decision-making performance, both if the human is assisted [17, 34] or is not assisted by a DSS [31].”
ORES (2019, neutral): “CSCW and HCI has also long studied human-in-the-loop systems [14, 37, 111], as well as explored implications of “algorithm-in-the-loop” socio-technical systems [44].”
To Trust or to Think (2021, background): “Prior studies in human-AI decision-making have shown the anchoring bias that occurs by presenting people with AI’s recommendation before allowing them to make a decision first [26].”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, methodology): “’s initial answer or only having the AI present an explanation if it disagreed with the human’s choice. We note that these approaches correspond to the Update and Feedback methods of Green & Chen [21], which were effective, albeit not in the complementary zone. However, by delaying display of th”
Algorithmic realism: expanding the boundaries of algorithmic thought (2020, background): “Yet such assessments typically overlook the ways in which these “fair” algorithms can lead to unfair social impacts, whether through biased uses by practitioners [2, 29, 55, 56], entrenching unjust policies [54], distorting deliberative processes [51], or shifting control of governance toward unacco”
Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, methodology): “On the other hand, similar to ethical concerns about static explanations raised in prior work [2, 20, 21], interactive explanations might reinforce”
Harnessing Explanations to Bridge AI and Humans (2020, background): “Some experiments in this vein have included predicting apartment prices [13], detecting deceptive online reviews [11], assessing social media toxicity [3], performing various artificial tasks [9], and recidivism prediction [5].”
Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making (2021, methodology): “1An example of task that may not be applicable to some decision making context is to examine human decision maker’s ability in detecting fairness problems of the AI model or utilizing the AI model more fairly, in the presence of explanations [23, 30].”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2022, background): “Human-AI performance generally only exceeds human-only performance [8, 26, 30, 55, 65, 78, 97]; stark performance differences between humans and AIs make it such that one would achieve the highest performance”