Explaining and Improving Model Behavior with k Nearest Neighbor Representations
By Nazneen Rajani, Ben Krause, Wen Yin, Tong Niu, Richard Socher, Caiming Xiong. 2020
39 citations recorded by the source corpus.
Interpretability techniques in NLP have mainly focused on understanding individual predictions using attention visualization or gradient-based saliency maps over tokens. We propose using k nearest neighbor (kNN) representations to identify training examples responsible for a model's predictions and obtain a corpus-level understanding of the model's behavior. Apart from interpretability, we show that kNN representations are effective at uncovering learned spurious associations, identifying mislabeled examples, and improving the fine-tuned model's performance. We focus on Natural Language Inference (NLI) as a case study and experiment with multiple datasets. Our method deploys backoff to kNN for BERT and RoBERTa on examples with low model confidence without any update to the model parameters. Our results indicate that the kNN approach makes the finetuned model more robust to adversarial inputs.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sameer Singh
- Mohit Bansal
- Yoav Goldberg
- Noah A. Smith
- Jacob Andreas
- Yulia Tsvetkov
- Matt Gardner
- Kelvin Guu
In Safety, Robustness & Interpretability
- Sameer Singh
- Byron C. Wallace
- Sarthak Jain
- Pouya Pezeshkpour
- Jacob Andreas
- Frederick Liu
- Tolga Bolukbasi
- Ian Tenney
Citations in the corpus
- Tracing Knowledge in Language Models Back to the Training Data (2022, methodology): “In this section, we present a formal description of the different influence methods we studied: gradient-based methods (Koh and Liang, 2017; Pruthi et al., 2020) and embedding-based methods (Rajani et al., 2020).”
- Combining Feature and Instance Attribution to Detect Artifacts (2021, methodology): “…their “impor-tance” for an output (Ribeiro et al., 2016; Lund-berg and Lee, 2017; Adebayo et al., 2018), and; (2) instance attribution , which retrieves training instances most responsible for a given prediction (Koh and Liang, 2017; Yeh et al., 2018; Rajani et al., 2020; Pezeshkpour et al., 2021).”
- An Empirical Comparison of Instance Attribution Methods for NLP (2021, methodology): “In this paper, we set out to evaluate and compare instance attri-bution methods, including relatively simple and efficient approaches (Rajani et al., 2020) in the context of NLP (Figure 1).”