Consistent Estimators for Learning to Defer to an Expert
By Hussein Mozannar, David A. Sontag. International Conference on Machine Learning, 2020
307 citations recorded by the source corpus.
Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert's decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/da71dbd6037f986af4ac28f7b792535ec1b44c6d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Daniel S. Weld
- Mohit Bansal
- Matt Gardner
- Luke S. Zettlemoyer
- Pang Wei Koh
- Tongshuang Sherry Wu
- Sameer Singh
Citations in the corpus
- Second opinion needed: communicating uncertainty in medical machine learning (2021, background): “Optimizing with respect to cost sensitive lost functions has been explored in many settings including binary predictions 55 – 58 , multiclass prediction 54 , class imbalance 53 , and deferring to experts 59 .”
- Selective Classification Can Magnify Disparities Across Groups (2020, methodology): “, 2019) and learn to abstain on examples human experts are more likely to get correct (Raghu et al., 2019; Mozannar and Sontag, 2020; De et al., 2020).”
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2020, background): “As recent work has suggested [4, 55, 59, 79], it may be useful to directly optimize for complementary behavior by accounting for the human behavior during training, who may have access to a diferent set of features [72].”