Causally motivated shortcut removal using auxiliary labels
By Maggie Makar, Ben Packer, Dan I. Moldovan, Davis W. Blalock, Yoni Halpern, A. D'Amour. 2021
84 citations recorded by the source corpus.
Shortcut learning, in which models make use of easy-to-represent but unstable associations, is a major failure mode for robust machine learning. We study a flexible, causally-motivated approach to training robust predictors by discouraging the use of specific shortcuts, focusing on a common setting where a robust predictor could achieve optimal \emph{iid} generalization in principle, but is overshadowed by a shortcut predictor in practice. Our approach uses auxiliary labels, typically available at training time, to enforce conditional independences implied by the causal graph. We show both theoretically and empirically that causally-motivated regularization schemes (a) lead to more robust estimators that generalize well under distribution shift, and (b) have better finite sample efficiency compared to usual regularization schemes, even when no shortcut is present. Our analysis highlights important theoretical properties of training techniques commonly used in the causal inference, fairness, and disentanglement literatures. Our code is available at https://github.com/mymakar/causally_motivated_shortcut_removal
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Houlsby
- Xiaohua Zhai
- Mario Lucic
- Quoc V. Le
- Katherine A. Heller
- Ghassen Jerfel
- Xuezhi Wang
- Percy Liang
In Safety, Robustness & Interpretability
- A. D'Amour
- Rory Sayres
- Jacob Eisenstein
- Katherine A. Heller
- David Sculley
- Kellie Webster
- Alex Beutel
- Xiaohua Zhai
Citations in the corpus
- Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “For example, Makar et al. (2021) and Veitch et al. (2021) show that regularization schemes that are designed to respect the causal structure of scenarios that are expected to be encountered in practice can produce models that generalize well in deployment; in particular, Makar et al. (2021)…”