Improving Transformer Models by Reordering their Sublayers

By Ofir Press, Noah A. Smith, Omer Levy. 2019

99 citations recorded by the source corpus.

Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers. Could ordering the sublayers in a different pattern lead to better performance? We generate randomly ordered transformers and train them with the language modeling objective. We observe that some of these models are able to achieve better performance than the interleaved baseline, and that those successful variants tend to have more self-attention at the bottom and more feedforward sublayers at the top. We propose a new transformer pattern that adheres to this property, the sandwich transformer, and show that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time. However, the sandwich reordering pattern does not guarantee performance gains across every task, as we demonstrate on machine translation models. Instead, we suggest that further exploration of task-specific sublayer reorderings is needed in order to unlock additional gains.

Subfields: Artificial Intelligence, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Optimization & Training

Citations in the corpus

Of Non-Linearity and Commutativity in BERT (2021, background): “Similarly, in [23] the authors exploit layer similarity to re-order the layers and improve performance, while in [7] and [29] they analyze layer redundancy to prune BERT with minimal performance degradation.”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, background): “Previous work also highlighted the importance of feed-forward layers in transformers (Press et al., 2020; Pulugundla et al., 2021; Xu et al., 2020).”
Shortformer: Better Language Modeling using Shorter Inputs (2021, methodology): “deviation of the Sandwich transformer (Press et al., 2020) and TransformerXL.”
Pay Attention when Required (2020, background): “Recent works, however, explored using random search [2] and evolutionary search [12, 13] for designing transformer models.”
Limits to Depth Efficiencies of Self-Attention (2020, background): “3 and 4 above captures the deﬁning self-attention characteristic of integrating the inputs with each other in a ﬂexible manner: The ReLU activation relaxation : Press et al. [2019] demonstrate that a “self-attention ﬁrst" BERT variant that ﬁrst performs all of the self-attention operations (eq.”
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention (2021, background): “Press et al. (2020) train a “self-attention first” network that first performs all of the self-attention operations consecutively, and only then performs all of the position-wise feed-forward operations.”
Understanding Robustness of Transformers for Image Classification (2021, methodology): “We then turn our attention to the action of the ViT models themselves, analyzing the evolution of information through the cascade of Transformer layers and the redundancies in their internal representations via correlation analysis and lesion studiesÐas has been done in the past for Transformers for”
A Survey of Transformers (2021, background): “Sandwich Transformer [99] explores reorganizing attention modules and FFN modules such that attention modules are mainly located in lower layers and FFN modules in upper layers.”