Reducing Transformer Depth on Demand with Structured Dropout

By Angela Fan, Edouard Grave, Armand Joulin. 2019

727 citations recorded by the source corpus.

Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality compared to training from scratch or using distillation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Beyond English-Centric Multilingual Machine Translation (2020, methodology): “To stabilize the training of deeper Transformers, we train with LayerDrop (Fan et al., 2019) 0.”
Training data-efficient image transformers & distillation through attention (2020, background): “, 2016), which facilitates the convergence of transformers, especially deep ones (Fan et al., 2019; 2020).”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, positive): “We also use layer drop (Fan et al., 2019), dropping entire layers at the FFN level.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “LARGE trains for 250k updates, BASE for 400k updates, and LARGE on LV-60k for 600k updates.”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “(2) Robustly trained transformers in which the model depth can be reduced at run-time, thereby decreasing inference time dynamically (Fan et al., 2020; Elbayad et al., 2020; Xin et al., 2020; Hou et al., 2020).”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Related work may include distillation (Sanh et al., 2019; Jiao et al., 2020; Wang et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al., 2019; Fan et al., 2019).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from diﬀerent aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “For comparison, we also fine-tune the English-only wav2vec 2.0 pretrained on Libri-Light which performs surprisingly well on this multilingual task but XLS-R outperforms it by 1.5% error rate on average.”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, methodology): “Additionally, LayerDrop [38] is applied with p = 5% .”
Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, methodology): “For all Transformer layers, we use dropout on the self-attention and on the FFN, and layer drop [28], dropping entire layers at the FFN level.”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “Dropout [30] and layer drop [31] are tuned and used to regularize each model.”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “, 2019), or entirely (Fan et al., 2019)) from a larger model.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “In structured pruning, architecture blocks are dropped, as in LayerDrop (Fan et al., 2019).”
Pre-Trained Models: Past, Present and Future (2021, background): “In (Fan et al., 2019), Transformer layers are selectively dropped during training, resulting in a more shallow model during inference.”