The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

By Elena Voita, Rico Sennrich, Ivan Titov. 2019

221 citations recorded by the source corpus.

We seek to understand how the representations of individual tokens and the structure of the learned feature space evolve between layers in deep neural networks under different learning objectives. We chose the Transformers for our analysis as they have been shown effective with various tasks, including machine translation (MT), standard left-to-right language models (LM) and masked language modeling (MLM). Previous work used black-box probing tasks to show that the representations learned by the Transformer differ significantly depending on the objective. In this work, we use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers and observe that the choice of the objective determines this process. For example, as you go from bottom to top layers, information about the past in left-to-right language models gets vanished and predictions about the future get formed. In contrast, for MLM, representations initially acquire information about the context around the token, partially forgetting the token identity and producing a more generalized token representation. The token identity then gets recreated at the top MLM layers.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Large Language Models

In Model Architectures

Citations in the corpus

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “It has repeatedly been shown that a MLM objective produces a better pretrained model for subsequent supervised finetuning (Devlin et al., 2018; Lample and Conneau, 2019; Voita et al., 2019; Raffel et al., 2020).”
Transformer Language Models without Positional Encodings Still Learn Positional Information (2022, neutral): “Finally, we observe that all models shed off a signiﬁcant amount of positional information in the ﬁnal layers, in line with the ﬁndings of Voita et al. (2019).”
Information-Theoretic Probing with Minimum Description Length (2020, background): “This is one more illustration of the general process explained in Voita et al. (2019a): the way representations evolve between layers is deﬁned by the training objective.”
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings (2020, result): “This is consistent with other work (Voita et al., 2019; Tenney et al., 2019): in the first layers the contextualization is too weak for high-quality alignments while the last layers are too specialized on the pretraining task (masked language modeling).”
Similarity Analysis of Contextual Word Representation Models (2020, methodology): “The vertical bands at layers 0 mean that many higher layers have a head that is very similar to a head from the ﬁrst layer, that is, a form of redundancy, which can explain why many heads can be pruned (Michel et al., 2019; Voita et al., 2019b; Kovaleva et al., 2019).”
Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases (2020, background): “[72] explain the change of vector representations among layers based on the compression/prediction trade-off perspective.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “In an analysis of text-based Transformers trained with a masked language model criterion [269] observed a similar compression plus reconstruction pattern.”
Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023, result): “Our analysis also relates to studies of the prediction process in LMs (Voita et al., 2019; Tenney et al., 2019).”
A Survey of Deep Learning for Scientific Discovery (2020, background): “Note however that the precise eﬀects of transfer learning are not yet fully understood, and an active research area [116, 184, 266, 159, 143, 181, 235] looks at investigating its exact properties.”
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation (2020, background): “A growing body of research is dedicated to the analyses of attention mechanisms and the interpretation of weights and connections (Raganato and Tiedemann, 2018; Tang et al., 2018; Mareˇcek and Rosa, 2019; Voita et al., 2019a).”
Transfusion: Understanding Transfer Learning for Medical Imaging (2019, methodology): “A recent tool that effectively overcomes these challenges is (Singular Vector) Canonical Correlation Analysis, (SV)CCA [22, 19], which has been used to study latent representations through training, across different models, alternate training objectives, and other properties [22, 19, 25, 18, 8, 17, ”
Staying True to Your Word: (How) Can Attention Become Explanation? (2020, methodology): “The last model we propose is inspired by re-sults in (Voita et al., 2019), where we introduce the masked language modelling objective (Devlin et al., 2019), in which input tokens from a sequence are masked at random.”
What Happens To BERT Embeddings During Fine-tuning? (2020, methodology): “Voita et al. (2019a) used a form of canonical correlation analysis (PW-CCA; Morcos et al., 2018) to study the layer-wise evolution of representations, while Saphra and Lopez (2019) explored how these representations evolve during training.”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, background): “Previous works mostly focused on analyzing the evolution of hidden representations across layers (Voita et al., 2019), and probing the model with target tasks (Yang et al., 2020; Clark et al., 2019; Tenney et al., 2019; Saphra and Lopez, 2019).”