Are Sixteen Heads Really Better than One?

By Paul Michel, Omer Levy, Graham Neubig. 2019

1,419 citations recorded by the source corpus.

Attention is a powerful and ubiquitous mechanism for allowing neural models to focus on particular salient pieces of information by taking their weighted average when making predictions. In particular, multi-headed attention is a driving force behind many recent state-of-the-art NLP models such as Transformer-based MT models and BERT. These models apply multiple attention mechanisms in parallel, with each attention "head" potentially focusing on different parts of the input, which makes it possible to express sophisticated functions beyond the simple weighted average. In this paper we make the surprising observation that even if models have been trained using multiple heads, in practice, a large percentage of attention heads can be removed at test time without significantly impacting performance. In fact, some layers can even be reduced to a single head. We further examine greedy algorithms for pruning down models, and the potential speed, memory efficiency, and accuracy improvements obtainable therefrom. Finally, we analyze the results with respect to which parts of the model are more reliant on having multiple heads, and provide precursory evidence that training dynamics play a role in the gains provided by multi-head attention.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “It also may be possible to remove entire attention heads in addition to zeroing out weights (Michel et al., 2019; Voita et al., 2019).”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “Recent studies (Kovaleva et al., 2019; Michel et al., 2019; Voita et al., 2019) demonstrate that there is redundancy in PLMs.”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, positive): “…giving easy access to the inner representations of these models, notably the hidden states, the attention weights or heads importance as deﬁned in Michel et al. (2019) and (ii) providing diﬀerent models in a uniﬁed API to prevent overﬁtting to a speciﬁc architecture (and set of pretrained…”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “eorts in the NLU literature for distilling [15, 33], compressing [40], and pruning [20] BERT can be instrumental in narrowing this gap, they generally achieve signicantly smaller speedups than our redesigned architecture for IR, due to their generic nature, and more aggressive optimizations oen c”
Towards Robust Vision Transformer (2021, background): “Following the popularity of transformers in Natural Language Processing (NLP) applications, e.g., BERT [8] and GPT [30], there has sparked particular interest in investigating whether transformer can be a primary backbone for computer vision applications previously dominated by Convolutional Neural ”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Related work may include distillation (Sanh et al., 2019; Jiao et al., 2020; Wang et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al., 2019; Fan et al., 2019).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “l., 2018), concerned with investigating the inner working of large-scale pretrained models and trying to build a science on top of these empirical results. Some examples include Tenney et al. (2019), Michel et al. (2019), Clark et al. (2019b). Transformers aims at facilitating and increasing the sco”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “Similarly, various papers have shown evidence that Transformers are over-parameterized, especially that most self-attention heads can be dropped at test time (Michel et al., 2019; Voita et al., 2019).”
Of Non-Linearity and Commutativity in BERT (2021, background): “While existing work has focused on analyzing the self-attention heads and the impact of model depth [14], [20], [34], we direct our investigation towards the FFN blocks of BERT.”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “, 2020), in which a smaller model is trained to imitate the larger one, or by model pruning (Fan et al., 2020; Michel et al., 2019), in which superfluous parameters are identified and discarded from the original model.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “More sophisticated approaches induce structured sparsity, which removes full blocks (Michel et al., 2019; Voita et al., 2019; Dodge et al., 2019b).”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “Less computation for ‘easy’ sub-words, more for others: One approach to reducing the amount of computation utilized by a model is to utilize structured pruning based approaches, which prune out layers (partially (Voita et al., 2019; Michel et al., 2019), or entirely (Fan et al.”
What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Michel et al. (2019) similarly show that many of BERT’s attention heads can be pruned.”
Efficient Transformers: A Survey (2020, background): “An example is to prune attention heads during inference (Voita et al., 2019; Michel et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “To mitigate that, Michel et al. (2019) prune heads in the order set by a proxy importance score, and Voita et al. (2019) fine-tune the pretrained Transformer with a regularized objective that has the head-disabling effect.”