Interpreting Neural Networks with Nearest Neighbors

By Eric Wallace, Shi Feng, Jordan L. Boyd-Graber. 2018

57 citations recorded by the source corpus.

Local model interpretation methods explain individual predictions by assigning an importance value to each input feature. This value is often determined by measuring the change in confidence when a feature is removed. However, the confidence of neural networks is not a robust measure of model uncertainty. This issue makes reliably judging the importance of the input features difficult. We address this by changing the test-time behavior of neural networks using Deep k-Nearest Neighbors. Without harming text classification accuracy, this algorithm provides a more robust uncertainty metric which we use to generate feature importance values. The resulting interpretations better align with human perception than baseline methods. Finally, we use our interpretation method to analyze model predictions on dataset annotation artifacts.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Deep Weighted Averaging Classifiers (2018, methodology): “[55] apply the method from Papernot and McDaniel to the problem of text classification and explore the implications for interpretability.”
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering (2018, background): “This simulates how model predictions change when a particular word’s embedding is set to the zero vector—it approximates word removal (Ebrahimi et al., 2018; Wallace et al., 2018).”
How Case-Based Reasoning Explains Neural Networks: A Theoretical Analysis of XAI Using Post-Hoc Explanation-by-Example from a Survey of ANN-CBR Twin-Systems (2019, methodology): “Another approach tries to map the layers of a DNN onto particular exemplar cases using Deep k-Nearest neighbors (DkNN) [71, 91, 92].”
Evaluating Explanation Without Ground Truth in Interpretable Machine Learning (2019, background): “Commonly, this property is evaluated on those explanations generated from search based methodologies [37], instead of those obtained from gradient based [33] or perturbation based [31] methodologies.”
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models (2019, methodology): “This is useful for aggregating interpretation results, e.g., as in Feng et al. (2018) and Wallace et al. (2018).”
Misleading Failures of Partial-input Baselines (2019, methodology): “For both the hypothesis-only and hypothesis-plus-one-word models, we follow Papernot and McDaniel (2018) and Wallace et al. (2018) and retrieve training examples using nearest neighbor search in the ﬁnal BERT representation space.”
A Survey of the State of Explainable AI for Natural Language Processing (2020, background): “Local Post-Hoc (11) feature importance first derivative saliency, example driven saliency 5 (Wallace et al., 2018; Ross et al., 2017)”
On Interpretability of Artificial Neural Networks: A Survey (2020, methodology): “As shown in Figure 12, E. Wallace et al. [181] employed the k-nearest neighbor algorithm to obtain the most similar cases for the query case in the feature space and then computed the percentage of the nearest neighbors belonging to the expected class as a measure for interpretability, suggesting…”