Revealing the Dark Secrets of BERT

By Olga Kovaleva, Alexey Romanov, Anna Rogers, Anna Rumshisky. 2019

635 citations recorded by the source corpus.

BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on the interpretation of self-attention, which is one of the fundamental underlying components of BERT. Using a subset of GLUE tasks and a set of handcrafted features-of-interest, we propose the methodology and carry out a qualitative and quantitative analysis of the information encoded by the individual BERT’s heads. Our findings suggest that there is a limited set of attention patterns that are repeated across different heads, indicating the overall model overparametrization. While different heads consistently use the same attention patterns, they have varying impact on performance across different tasks. We show that manually disabling attention in certain heads leads to a performance improvement over the regular fine-tuned BERT models.

Subfields: Artificial Intelligence, Information Systems, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “large margin. This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very dierent from pre-training tasks. 4.4 Visualization Similar to [17], we observe several patterns in the attention maps of the UNITER model, as shown in Fig.2. Note ”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “domain-specific tasks (Kovaleva et al., 2019).”
Rethinking Attention with Performers (2020, neutral): “These patterns are consistent with the patterns found in Transformer models trained on natural language (Kovaleva et al., 2019).”
Longformer: The Long-Document Transformer (2020, background): “Sliding Window Given the importance of local context (Kovaleva et al., 2019), our attention pattern employs a ﬁxed-size window attention surrounding each token.”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “Furthermore, retrieving a block instead of a single token allows the attention to also access the local context around the token which may be more accommodating of observed classic attention patterns [20].”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “Combining this fact with the observation that BERT is over-parameterized (for example, Kovaleva et al. [2019]) leads to the interesting question of whether smaller models might be just as effective, particularly if limited to a specific task such as text ranking.”
Knowledge Neurons in Pretrained Transformers (2021, methodology): “It is motivated by FFNs’s analogy with the self-attention mechanism (as described in Section 2), because self-attention scores are usually used as a strong attribution baseline (Kovaleva et al., 2019; Voita et al., 2019; Hao et al., 2021).”
Of Non-Linearity and Commutativity in BERT (2021, background): “In this field, a number of studies look into attention distributions in order to understand transformers’ decisions and their internal dynamics [4], [11], [16].”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “For instance, some heads focus on adjacent tokens while others align with syntactic properties (Kovaleva et al., 2019; Hoover et al., 2019; Clark et al., 2019; Vig and Belinkov, 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “At the moment, we know that the knowledge represented in BERT does not necessarily get used in down-stream tasks (Kovaleva et al., 2019).”
PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers (2020, background): “Along with these quantitative advances have come deeper understanding of the learned hidden representations which power transformers (Kovaleva et al., 2019; Voita et al., 2019; Clark et al., 2019; Ethayarajh, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “Visualization techniques have been used to convey the structure and properties of attention in Transformers [31, 40, 80, 82].”
Pre-Trained Models: Past, Present and Future (2021, background): “Furthermore, Kovaleva et al. (2019) conduct a qualitative and quantitative analysis of the information encoded by PTMs ’ heads.”
Information-Theoretic Probing with Minimum Description Length (2020, background): “…uses to encode information, such as attention weights (Voita et al., 2018; Raganato and Tiede-mann, 2018; Voita et al., 2019b; Clark et al., 2019; Kovaleva et al., 2019) or individual neurons (Karpa-thy et al., 2015; Pham et al., 2016; Bau et al., 2019), (ii) looking at model predictions using…”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, methodology): “As recent studies (Kovaleva et al., 2019) have shown redundancy in pre-training models, it is useful to design a onesize-fits-all model that caters to samples with varying complexity and gains computational efficacy with the least loss of accuracy.”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “There has been several studies that investigate latent knowledge encoded in pre-trained language models [33,7,25,16].”
What Does BERT with Vision Look At? (2020, background): “, 2019) while another studies the attention mechanism (Clark et al., 2019; Voita et al., 2019; Kovaleva et al., 2019).”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “We now know that most Trans-former heads and even layers can be pruned (Voita et al., 2019; Michel et al., 2019; Kovaleva et al., 2019), but it is not clear whether that is due to redundant weights, or to some parts of the model simply being “inactive” (Zhang et al., 2019).”
Implicit Representations of Meaning in Neural Language Models (2021, background): “Within the realm of semantics, existing work has identi-ﬁed representations of word meaning (e.g., ﬁne-grained word senses; Wiedemann et al. 2019) and predicate–argument structures like frames and semantic roles (Kovaleva et al., 2019).”