Interpretability and Analysis in Neural NLP

By Yonatan Belinkov, Sebastian Gehrmann, Ellie Pavlick. 2020

78 citations recorded by the source corpus.

While deep learning has transformed the natural language processing (NLP) field and impacted the larger computational linguistics community, the rise of neural networks is stained by their opaque nature: It is challenging to interpret the inner workings of neural network models, and explicate their behavior. Therefore, in the last few years, an increasingly large body of work has been devoted to the analysis and interpretation of neural network models in NLP. This body of work is so far lacking a common framework and methodology. Moreover, approaching the analysis of modern neural networks can be difficult for newcomers to the field. This tutorial aims to fill this gap and introduce the nascent field of interpretability and analysis of neural networks in NLP. The tutorial will cover the main lines of analysis work, such as structural analyses using probing classifiers, behavioral studies and test suites, and interactive visualizations. We will highlight not only the most commonly applied analysis methods, but also the specific limitations and shortcomings of current approaches, in order to inform participants where to focus future efforts.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Improving language models by retrieving from trillions of tokens (2021, background): “Finally, samples from large models are difficult to interpret, making mitigating these issues all the more challenging (Belinkov et al., 2020; Jain and Wallace, 2019).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Belinkov et al. (2020) discuss how their probing experiments can guide the selection of which machine translation models to use when translating speciﬁc languages.”
Predicting Inductive Biases of Pre-Trained Models (2021, background): “Currently, the question of how models’ internal representations (measured by probing classiﬁers) inﬂuence model behavior (measured by challenge sets) remains open (Belinkov & Glass, 2019; Belinkov et al., 2020).”
A Review on Language Models as Knowledge Bases (2022, background): “In its most basic form, one type of probing is done by way of the use of a simple linear classiﬁer to associate internal representations with external properties (Belinkov et al., 2020).”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “As mentioned, this area of research is quite large and Belinkov et al. [14] cover behavioral probes in detail.”
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020, methodology): “Despite the recent explosion of work on model understanding and evaluation (e.g. Belinkov et al., 2020; Linzen et al., 2019; Ribeiro et al., 2020), there is no “silver bullet” for analysis.”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Belinkov et al. (2020) discuss how their probing experiments can guide the selection of which machine translation models to use when translating speciﬁc languages.”