Debugging Tests for Model Explanations

By Julius Adebayo, Michael Muelly, Ilaria Liccardi, Been Kim. 2020

208 citations recorded by the source corpus.

We investigate whether post-hoc model explanations are effective for diagnosing model errors--model debugging. In response to the challenge of explaining a model's prediction, a vast array of explanation methods have been proposed. Despite increasing use, it is unclear if they are effective. To start, we categorize \textit{bugs}, based on their source, into:~\textit{data, model, and test-time} contamination bugs. For several explanation methods, we assess their ability to: detect spurious correlation artifacts (data contamination), diagnose mislabeled training examples (data contamination), differentiate between a (partially) re-initialized model and a trained one (model contamination), and detect out-of-distribution inputs (test-time contamination). We find that the methods tested are able to diagnose a spurious background bug, but not conclusively identify mislabeled training examples. In addition, a class of methods, that modify the back-propagation algorithm are invariant to the higher layer parameters of a deep network; hence, ineffective for diagnosing model contamination. We complement our analysis with a human subject study, and find that subjects fail to identify defective models using attributions, but instead rely, primarily, on model predictions. Taken together, our results provide guidance for practitioners and researchers turning to explanations as tools for model debugging.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, background): “It enables the former to be used for uncertainty aware tasks, such as OOD detection (Daxberger & Hernández-Lobato, 2019), continual learning (Nguyen et al., 2018), active learning (Depeweg et al., 2018), and Bayesian optimization (Springenberg et al., 2016).”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “Still, saliency mapping can be used to detect spurious bugs or artefacts in the input feature space (although they do not always detect them in ‘contaminated’ models 323,324 ).”
Benchmarking and survey of explanation methods for black box models (2021, background): “In (Adebayo et al. 2018, 2020), some weaknesses of smoothgrad are shown: people tend to evaluate SMs on what they are expected to see.”
Acquisition of chess knowledge in AlphaZero (2021, background): “that explanations from a trained and an untrained model are visually indistinguishable and [37] also hinted that the explanations do not seem to be useful for debugging typical ML model problems.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “[4] summarized the uses of interpretation algorithms for model debugging, i.”