Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure

By Dieuwke Hupkes, Willem H. Zuidema. 2017

304 citations recorded by the source corpus.

We investigate how neural networks can learn and process languages with hierarchical, compositional semantics. To this end, we define the artificial task of processing nested arithmetic expressions, and study whether different types of neural networks can learn to compute their meaning. We find that recursive neural networks can find a generalising solution to this problem, and we visualise this solution by breaking it up in three steps: project, sum and squash. As a next step, we investigate recurrent neural networks, and show that a gated recurrent unit, that processes its input incrementally, also performs very well on this task. To develop an understanding of what the recurrent network encodes, visualisation techniques alone do not suffice. Therefore, we develop an approach where we formulate and test multiple hypotheses on the information encoded and processed by the network. For each hypothesis, we derive predictions about features of the hidden state representations at each time step, and train 'diagnostic classifiers' to test those predictions. Our results indicate that the networks follow a strategy similar to our hypothesised 'cumulative strategy', which explains the high accuracy of the network on novel expressions, the generalisation to longer expressions than seen in training, and the mild deterioration with increasing length. This is turn shows that diagnostic classifiers can be a useful technique for opening up the black box of neural networks. We argue that diagnostic classification, unlike most visualisation techniques, does scale up from small networks in a toy domain, to larger and deeper recurrent networks dealing with real-life data, and may therefore contribute to a better understanding of the internal dynamics of current state-of-the-art models in natural language processing.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public.”
Locating and Editing Factual Associations in GPT (2022, background): “…are encoded in internal model representations, most commonly by training a probing classiﬁer to predict said properties from the representations (Ettinger et al., 2016; Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018; Belinkov et al., 2017; Belinkov & Glass, 2019, inter alia).”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, neutral): “, 2016) and diagnostic classification (Hupkes et al., 2018)).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Structural analyses aim to shed light on the internal structure of a neural model, for example through probing classiﬁers (Conneau et al., 2018; Hupkes et al., 2018; Adi et al., 2017) that predict linguistic properties using representations from trained models.”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “A similar method has been used to analyze hierarchical structure in neural networks trained on arithmetic expressions (Veldhoen et al., 2016; Hupkes et al., 2018).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “…of recurrent networks and related models by using visualization, input/hidden representation deletion techniques or by looking at the word-by-word behaviour of the network (e.g., Nagamine et al., 2015; Hupkes et al., 2017; Li et al., 2016; Linzen et al., 2016; Kàdàr et al., 2017; Li et al., 2017).”
Designing and Interpreting Probes with Control Tasks (2019, methodology): “Probing papers tend to acknowledge this uncertainty, putting accuracies in context using random representation baselines (Zhang and Bowman, 2018) and careful task design (Hupkes et al., 2018).”
A Structural Probe for Finding Syntax in Word Representations (2019, methodology): “Such methods work by specifying a probe (Conneau et al., 2018; Hupkes et al., 2018), a supervised model for finding information in a representation.”
Colorless Green Recurrent Networks Dream Hierarchically (2018, methodology): “On the one hand, we plan to adapt methods to inspect information ﬂow across RNN states (e.g., Hupkes et al., 2017).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “( § 6) We use probing models 1 (Shi et al., 2016b; Adi et al., 2017; Hupkes et al., 2018; Belinkov and Glass, 2019) to analyze the linguistic information within CWR s. Concretely, we generate features for words from pretrained contextualiz-ers and train a model to make predictions from those…”
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information (2018, methodology): “To this end, we use diagnostic classifiers (DCs, Hupkes et al., 2018).”
What Does BERT Learn about the Structure of Language? (2019, background): “Probing (or diagnostic) tasks (Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018) help in unearthing the linguistic features possibly encoded in neural models.”
Open Sesame: Getting inside BERT’s Linguistic Knowledge (2019, background): “This is similar to the concept of diagnostic classification by Hupkes et al. (2018); Giulianelli et al. (2018). the cat will sleep (BERT) ↓ ↓ ↓ ↓ e1 e2 e3 e4 (Classifier) ↓ ↓ ↓ ↓ ŷ1 ŷ2 ŷ3 ŷ4 Each input sentence is tokenized and processed by BERT, and the resulting embeddings {ei} are…”
The emergence of number and syntax units in LSTM language models (2019, neutral): “Another research line studied linguistic processing in neural networks through ‘diagnos-tic classiﬁers’, that is, classiﬁers trained to predict a certain property from network activations (e.g., Gelderloos and Chrupała, 2016; Adi et al., 2017; Alain and Bengio, 2017; Hupkes et al., 2018).”
Investigating Transferability in Pretrained Language Models (2020, methodology): “Recent work on understanding deep NLP models has centered on probing, a methodology that involves training classifiers for different tasks on model representations (Alain and Bengio, 2016; Conneau et al., 2018; Hupkes et al., 2018; Liu et al., 2019; Tenney et al., 2019a,b; Goldberg, 2019; Hewitt and”
Diagnostic classification and symbolic guidance to understand and improve recurrent neural networks (2017, methodology): “To do so, we will employ the method of diagnostic classification [Hupkes et al., 2017], in which hypotheses about what is encoded in the hidden state of a neural network are tested by training additional neural classifiers to predict features following from these hypotheses.”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “How well the probe learns a given signal can be seen as a proxy for linguistic knowledge encoded in the representations (Hupkes et al., 2018).”