Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative

By A. Levy, Monica Agrawal, Arvind Satyanarayan, David A. Sontag. 2021

94 citations recorded by the source corpus.

Automated decision support can accelerate tedious tasks as users can focus their attention where it is needed most. However, a key concern is whether users overly trust or cede agency to automation. In this paper, we investigate the effects of introducing automation to annotating clinical texts — a multi-step, error-prone task of identifying clinical concepts (e.g., procedures) in medical notes, and mapping them to labels in a large ontology. We consider two forms of decision aid: recommending which labels to map concepts to, and pre-populating annotation suggestions. Through laboratory studies, we find that 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement. However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations. Our findings inform how systems and algorithms should be designed to mitigate the observed issues.

Subfields: Artificial Intelligence, Health Informatics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Natural Language Processing

Citations in the corpus

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “In a collaboration, AI can play various roles, such as casual creators that encourage exploration [26] or assistants that compensate for humanweaknesses [41, 74].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “Model uncertainty Classification confidence (or probability) [10, 13, 21, 22, 43, 60, 68, 73, 86, 87, 144, 155], flag low confidence [90], uncertainty distribution (regression) [101] Local feature importance Coefficients [16, 28, 39, 43, 49, 55, 80, 81, 86, 87, 94, 106, 111, 125, 128, 143], attentio”
Large language models are few-shot clinical information extractors (2022, methodology): “For each task, a set of examples was jointly annotated to establish an annotation schema, each annotator then independently labeled the same set of 105 examples using PRAnCER software (Levy et al., 2021), and the two sets were then merged via joint manual adjudication.”