Learning under selective labels in the presence of expert consistency
By Maria De-Arteaga, Artur Dubrawski, Alexandra Chouldechova. 2018
42 citations recorded by the source corpus.
We explore the problem of learning under selective labels in the context of algorithm-assisted decision making. Selective labels is a pervasive selection bias problem that arises when historical decision making blinds us to the true outcome for certain instances. Examples of this are common in many applications, ranging from predicting recidivism using pre-trial release data to diagnosing patients. In this paper we discuss why selective labels often cannot be effectively tackled by standard methods for adjusting for sample selection bias, even if there are no unobservables. We propose a data augmentation approach that can be used to either leverage expert consistency to mitigate the partial blindness that results from selective labels, or to empirically validate whether learning under such framework may lead to unreliable models prone to systemic discrimination.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Alexandra Chouldechova
- Amanda Coston
- Elliot Creager
- Shira Mitchell
- David Madras
- Solon Barocas
- Silvia Chiappa
- Luca Oneto
Citations in the corpus
- Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “Another option is to trust previous decisions: assume that regions of the input variables where no defendants were released (no d = 0) are regions where all defendants would have commited crimes if released ( y = 1) [29].”
- Algorithmic Fairness: Choices, Assumptions, and Definitions (2021, background): “Another option is to trust previous decisions: Assume that regions of the covariates where no defendants were released are regions where all defendants would have committed crimes if released (De-Arteaga et al. 2018).”
- Counterfactual risk assessments, evaluation, and fairness (2019, result): “Our approach contrasts to prior work which used placement out-of-home as the outcome [8, 11].”
- Fairness through Causal Awareness: Learning Causal Latent-Variable Models for Biased Data (2018, background): “Some conceptually similar work has looked at the “selective labels” problem [9, 29], where only a biased selection of the data has labels available.”