Analysis Methods in Neural Language Processing: A Survey

By Yonatan Belinkov, James R. Glass. 2018

652 citations recorded by the source corpus.

The field of natural language processing has seen impressive progress in recent years, with neural network models replacing many of the traditional systems. A plethora of new models have been proposed, many of which are thought to be opaque compared to their feature-rich counterparts. This has led researchers to analyze, interpret, and evaluate neural networks in novel and more fine-grained ways. In this survey paper, we review analysis methods in neural language processing, categorize them according to prominent research trends, highlight existing limitations, and point to potential directions for future work.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “See Belinkov and Glass (2019) for a recent survey of analysis methods.”
Are NLP Models really able to Solve Simple Math Word Problems? (2021, methodology): “Challenge Sets for NLP tasks have been proposed most notably for NLI and machine translation (Belinkov and Glass, 2019; Nie et al., 2020; Ribeiro et al., 2020).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public.”
oLMpics-On What Language Model Pre-training Captures (2019, background): “…often linear, randomly initialized probe, to determine whether the representations hold relevant information (Ettinger et al., 2016; Adi et al., 2016; Belinkov and Glass, 2019; He-witt and Manning, 2019; Wallace et al., 2019; Rozen et al., 2019; Warstadt et al., 2019; Richard-son et al., 2019).”
How Can We Know What Language Models Know? (2019, methodology): “This is conceptually similar to query expansion techniques used in information retrieval that reformulate a given query to improve retrieval performance (Carpineto and Romano, 2012).”
Alignment of Language Agents (2021, methodology): “For a survey of interpretability/analysis methods in neural NLP see Belinkov and Glass (2019).”
Locating and Editing Factual Associations in GPT (2022, background): “…are encoded in internal model representations, most commonly by training a probing classiﬁer to predict said properties from the representations (Ettinger et al., 2016; Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018; Belinkov et al., 2017; Belinkov & Glass, 2019, inter alia).”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Probes also only give correlational evidence, rather than causal evidence (Belinkov and Glass, 2019).”
Editing Factual Knowledge in Language Models (2021, background): “One cannot easily access and interpret their computation and memories (Ribeiro et al., 2016; Belinkov and Glass, 2019; Voita et al., 2019; De Cao et al., 2020), and thus, modifying the knowledge is also a challenging problem.”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “phonetic information in layers of a speech recognition systems (Belinkov and Glass 2017) partly motivated Krishna, Toshniwal, and Livescu (2019) to propose multi-task learning with phonetic supervision on intermediate layers.”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “Many papers have found neural models (especially PLMs) to naturally encode syntax (Linzen et al., 2016; Belinkov et al., 2017; Marvin and Linzen, 2018; Belinkov and Glass, 2019; Goldberg, 2019; Hewitt and Manning, 2019).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “As pointed out by Belinkov and Glass (2019), probing classiﬁers only yield a correlational measure between a model’s representations and an ex-ternal linguistic property, and are thus not causally connected to the model’s predictions.”
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020, background): “Belinkov and Glass (2019) note beneﬁts of this approach, such as systematic control over data, as well as drawbacks, such as small scale and lack of resemblance to “real” data.”
Designing and Interpreting Probes with Control Tasks (2019, methodology): “Applying dropout, the most popular probing regularization method (Adi et al., 2017; Belinkov and Glass, 2019; Şahin et al., 2019; Kim et al., 2019; Elloumi et al., 2018; Belinkov and Glass, 2017; Belinkov et al., 2018) does not consistently lead to high-accuracy, high-selectivity MLP probes across a”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “( § 6) We use probing models 1 (Shi et al., 2016b; Adi et al., 2017; Hupkes et al., 2018; Belinkov and Glass, 2019) to analyze the linguistic information within CWR s. Concretely, we generate features for words from pretrained contextualiz-ers and train a model to make predictions from those…”
What Does BERT Learn about the Structure of Language? (2019, background): “BERT outperforms previous state-of-the-art models in the eleven NLP tasks in the GLUE benchmark (Wang et al., 2018) by a signiﬁcant margin.”
A Multiscale Visualization of Attention in the Transformer Model (2019, background): “An advantage of using attention is that it can help interpret a model by showing how the model assigns weight to different input elements (Bahdanau et al., 2015; Belinkov and Glass, 2019), although its value in explaining individual predictions may be limited (Jain and Wallace, 2019).”
Analyzing the Structure of Attention in a Transformer Language Model (2019, background): “An advantage of using attention is that it can help interpret the model by showing how the model attends to different parts of the input (Bahdanau et al., 2015; Belinkov and Glass, 2019).”