Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

By Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, Yoav Goldberg. 2016

583 citations recorded by the source corpus.

There is a lot of research interest in encoding variable length sentences into fixed length vectors, in a way that preserves the sentence meanings. Two common methods include representations based on averaging word vectors, and representations based on the hidden states of recurrent neural networks such as LSTMs. The sentence vectors are used as features for subsequent machine learning tasks or for pre-training in the context of deep learning. However, not much is known about the properties that are encoded in these sentence representations and about the language information they capture. We propose a framework that facilitates better understanding of the encoded representations. We define prediction tasks around isolated aspects of sentence structure (namely sentence length, word content, and word order), and score representations by the ability to train a classifier to solve each prediction task when using the representation as input. We demonstrate the potential contribution of the approach by analyzing different sentence representation mechanisms. The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector's dimensionality on the resulting representations.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Neural Network Acceptability Judgments (2018, background): “This contributes to a growing effort to test ANNs’ ability to make finegrained grammatical distinctions (Linzen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Ettinger et al., 2018; Marvin and Linzen, 2018).”
An efficient framework for learning sentence representations (2018, methodology): “Alternative fine-grained evaluation tasks such as identifying word appearance and word order were proposed in Adi et al. (2017). Although this provides some useful insight about the representations, these tasks focus on the syntactic aspects of a sentence.”
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “Outside the NLI probing literature, multiple projects have used classiﬁcation tasks to understand what linguistic and/or structural information is captured by vector encodings (e.g. Adi et al., 2017; Ettinger et al., 2018; Conneau et al., 2018).”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, methodology): “We measure this information in two ways: a) with diagnostic classiﬁers (Adi et al., 2017); and b) with mutual information (MI; Noshad et al., 2018).”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, methodology): “ditional tasks as the consensus on the best evaluation for sentence embeddings evolves. In particular, tasks that probe for speciﬁc linguistic properties of the sentence embeddings (Shi et al., 2016; Adi et al., 2017) are interesting directions towards understanding how the encoder understands langu”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “Similarly, Adi et al. (2016) probe representations extracted by sequence autoencoders, word embedding averages, and skip-thought vectors with a multi-layer perceptron (MLP) classifier to study whether sentence characteristics such as length, word content and word order are encoded. To generalize acr”
Pointer Sentinel Mixture Models (2016, background): “As we increase both the number of timesteps and the window of words for the pointer component to consider, the RNN hidden state by itself isn’t guaranteed to accurately recall the identity or order of words it has recently seen (Adi et al., 2016).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public.”
oLMpics-On What Language Model Pre-training Captures (2019, background): “…a simple, often linear, randomly initialized probe, to determine whether the representations hold relevant information (Ettinger et al., 2016; Adi et al., 2016; Belinkov and Glass, 2019; Hewitt and Manning, 2019; Wallace et al., 2019; Rozen et al., 2019; Peters et al., 2018b; Warstadt et…”
Locating and Editing Factual Associations in GPT (2022, background): “…are encoded in internal model representations, most commonly by training a probing classiﬁer to predict said properties from the representations (Ettinger et al., 2016; Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018; Belinkov et al., 2017; Belinkov & Glass, 2019, inter alia).”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “5 “fine-grained measurement of some of the information encoded in sentence embeddings” (Adi et al. 2017); “simple linguistic properties of sentences” (Conneau et al.”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, background): “One prominent methodology that attempts to shed light on those questions is probing (Conneau et al., 2018) (also known as auxilliary prediction [Adi et al., 2016] and diagnostic classification [Hupkes et al., 2018]).”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “Structural methods try to identify the information encoded in the model’s internal structure by using its representations to classify textual properties (Adi et al. 2017; Hupkes, Veldhoen, and Zuidema 2018; Conneau et al. 2018).”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Structural analyses aim to shed light on the internal structure of a neural model, for example through probing classiﬁers (Conneau et al., 2018; Hupkes et al., 2018; Adi et al., 2017) that predict linguistic properties using representations from trained models.”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “Shi et al. (2016) and Adi et al. (2017) introduced a more general approach, relying on the notion of what we will call probing tasks .”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, neutral): “This approach, which bears similarities with analysis methods presented by Adi et al. (2017) and Gelderloos and Chrupa la (2016), can be used to quantitatively test for the existence of feature detectors such as the inside-quote detectors found by Karpathy et al. (2015), but also can be extended to ”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Recent work has investigated this question by examining the outputs of language models on carefully chosen input sentences (Linzen et al., 2016) or examining the internal vector representations of the model through methods such as probing classiﬁers (Adi et al., 2017; Belinkov et al., 2017).”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, methodology): “The methodology of probing tasks (e.g. Adi et al., 2017; Ettinger et al., 2018) has been used to show that large LMs learn at least some information about phenomena such as English subject-verb agreement (Goldberg, 2019; Jawahar et al., 2019), constituent types, dependency labels, NER, and (core)…”