A Benchmark for Interpretability Methods in Deep Neural Networks

By Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, Been Kim. 2018

879 citations recorded by the source corpus.

We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Unlike prior work in computer vision (Hooker et al., 2019), we ﬁnd raw gradients to perform comparably to the input*gradient variant (Shrikumar et al., 2017).”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “, the reliability of existing methods is an active area of research [85] [86] [87] [88] [89] [90] [91] [92].”
Making deep neural networks right for the right scientific reasons by interacting with their explanations (2020, neutral): “If this is not the case, changes in accuracy might be due to artifacts of different data, rather than deficits of the model [62].”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, methodology): “Our AOPC metrics are similar in concept to ROAR (Hooker et al., 2019) except that we re-use an existing model as opposed to retraining for each fraction.”
Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models (2020, methodology): “To quantitatively evaluate the causal relevance of competing saliency maps, we adopt the evaluation scheme proposed in [27], consisting of masking out the contributing saliency pixels and then calculating the reduction in prediction score.”
Explainability in Graph Neural Networks: A Taxonomic Survey (2020, methodology): “We further study our proposed evaluationmetric Fidelity with a random designation of feature importance [86].”
Transformer Interpretability Beyond Attention Visualization (2020, methodology): “Evaluation settings For the visual domain, we follow the convention of reporting results for negative and positive perturbations, as well as showing results for segmentation, which can be seen as a general case of ”The Pointing-Game” [19].”
Selective Brain Damage: Measuring the Disparate Impact of Model Pruning (2019, background): “PIE could be used to surface atypical examples for further human inspection [48], choose not to classify certain examples when the model is uncertain [2, 10, 9, 11], or to aid interpretability as a case based reasoning tool to explain model behavior [41, 23, 4, 33].”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “In other cases, most notably input features , providing a functionally-grounded metric can be very challenging [3, 51, 53, 60, 135].”
Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics (2021, background): “…[82] Attribution-based explanations Monotonicity [80] Non-sensitivity [80,83], Sensitivity [84,85] Effective complexity [80] Remove and retrain [86] Recall of important features [33] Implementation invariance [85] Selectivity [87] Continuity [87] Sensitivity-n [88] Mutual information [80]…”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “However, since some salience methods are too sensitive to input changes to produce reliable results [64], [65], [66], thus attention-based methods [21], [56], [62], [67], [68] are more studied than saliency-based methods [57], [69] on locating visual objects.”
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020, background): “There is now a wealth of work on evaluating explanations of machine learning models (Ribeiro et al., 2016; Doshi-Velez and Kim, 2017; Hooker et al., 2019; Jacovi and Goldberg, 2020b).”