Depth-Adaptive Transformer

By Maha Elbayad, Jiatao Gu, Edouard Grave, Michael Auli. 2019

284 citations recorded by the source corpus.

State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well tuned baseline Transformer while using less than a quarter of the decoder layers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Signal Processing

Citations in the corpus

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Conditional Computation Conditional computation (Bengio et al., 2015; Elbayad et al., 2020) postulates that examples should be routed within the network by activating an input dependent sub-network.”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “These adaptive computation methods (e.g. Han et al., 2021; Sukhbaatar et al., 2019; Schuster et al., 2021; Scardapane et al., 2020; Bapna et al., 2020; Elbayad et al., 2019; Schwartz et al., 2020) aim to use less compute resources for easier inference steps.”
Dynamic Neural Networks: A Survey (2021, methodology): “Language Sa & Te Language modeling [11], [16], [118], [170], [172], Machine translation [16], [35], [36], Classification [64], [65], [174], Processing Sentiment analysis [168], [169], [171], [175], [176], Question answering [35], [63], [168], [171], [173]”
Controlling Computation versus Quality for Neural Sequence Models (2020, positive): “DAT requires explicit supervision from oracles (or implicit supervision from multiple softmax computations) to train halting classifiers, restricting their approach to specific applications (like decoders in sequence to sequence models).”
Pre-trained models for natural language processing: A survey (2020, background): “[185] proposed Depth-adaptive transformer for machine translation, which learns to predict how many decoding layers are required for a particular sequence or token.”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “In [4, 7, 6, 13, 29, 34], Transformerbased models with adaptive depth are proposed to dynamically select some of the Transformer layers during inference.”
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021, background): “Moreover, our method requires neither RL nor complicated distribution ﬁtting as in DAT, but uses a straightforward layer-wise certainty estimation, and achieves performance comparable with conﬁdence-based early exiting on classiﬁcation tasks.”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “tematically compare with five state-of-the-art halting mechanisms, covering both vision and NLP methods that tackle the dynamic inference problem from different perspectives: (i) adaptive computation time [17] as ACT reference applied on halting entire layers, (ii) confidence-based halting [31] that”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “Conditional computation has previously been successfully applied to adapt the amount of computation to the input in recurrent models (Graves, 2016) and transformers (Dehghani et al., 2019; Elbayad et al., 2020), or to significantly scale up model capacity by utilizing sparsely-gated Mixture-of-Exper”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, methodology): “Such methods often use small neural models to determine when to stop the execution process (Schwartz et al., 2020b; Elbayad et al., 2020; Hou et al., 2020; Xin et al., 2020, 2021; Li et al., 2021; Schuster et al., 2021).”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “A similar idea is also presented in Elbayad et al. (2020) and Liu et al. (2021b) where hidden states of some positions can be frozen and directly copied to upper layers, although the former is focused on generation and the latter is for classiﬁcation.”
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, background): “The idea of applying language heads directly to the hidden states of the middle layers, known as early exit (Teerapittayanon et al., 2016; Elbayad et al., 2020; Schuster et al., 2022), has proven to be effective even without special training process (Kao et al., 2020), as the residual connections…”
A Practical Survey on Faster and Lighter Transformers (2021, background): “[69] addressed this limitation with the DepthAdaptive Transformer (DAT), which applies different layers at every depth.”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, methodology): “, 2020), computational budget (Bapna et al., 2019; Elbayad et al., 2019) or simplifying the expert allocation and training regimen (Lewis et al.”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, methodology): “Specifically, these studies [24, 67, 87, 109, 132, 140, 141, 148, 150, 155, 157, 158, 168, 172] define a loss function for each of the classifier, and minimize the weighted sum of cross entropy losses per sample as follows: where 3) by treating the final classifier (last exit) as teacher model and…”
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024, unknown): “We build upon Elbayad et al. (2020) and set a scale that increases across layers, such as the scale at one layer is proportional to the sum of the scales of all previous layers: P This way, we penalize later layers with quadratically higher weight, as predicting in later layers is easier.”
Confident Adaptive Language Modeling (2022, background): “2) for convenience, following previous work [23; 70; 76].”