Do Transformer Modifications Transfer Across Implementations and Applications?

By Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan and 6 more. 2021

138 citations recorded by the source corpus.

The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, methodology): “1 TRANSFERABILITY OF RESULTS TO VISION TRANSFORMERS (VIT) Following our language experiments, and as per the advice of (Narang et al., 2021), in order to examine our DeepNarrow scaling strategy in another domain and check if the observations extends to the cases where Transformers are applied on oth”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “On the other hand, a great deal of work on improving existing architectures has seen relatively little adoption (Narang et al., 2021); adopting some of these recommended practices could yield a significantly better model.”
A Review of Sparse Expert Models in Deep Learning (2022, background): “Fedus et al. (2021); Narang et al. (2021) compared the pre-training perplexity versus fine-tuning performance for dense and sparse models.”
UL2: Unifying Language Learning Paradigms (2022, neutral): “This is also in light of results such as (Narang et al., 2021).”
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, background): “, 2021), the relative value of advances in model architectures (Narang et al., 2021; Tay et al., 2022), the use of language models for generation tasks (Bender et al.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “Past studies on architectures and objectives for language models [e.g. Narang et al., 2021, Raffel et al., 2020] have focused mainly on the transfer learning setting, with models that were orders of magnitude smaller than the current state-of-the-art.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “These modifications were benchmarked and shown to be effective in T5 [49].”
Sparse is Enough in Scaling Transformers (2021, background): “Table 1 in [31] shows the same final loss for ReLU and GeLU on the C4 dataset, though, so in this work for simplicity, we focus on ReLU.”
The Efficiency Misnomer (2021, background): “Note that many works in the literature employ compute-matched comparisons for comparing sparse models (Narang et al., 2021; Fedus et al., 2021; Lample et al., 2019).”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “, 2018), such as Adafactor (instead of Adam) for optimization, inverse square root learning schedule, and pre-layer (as opposed to post-layer) normalization (Narang et al., 2021).”
A Survey of Large Language Models (2023, unknown): “Gopher [59] and Chinchilla [34], RMS Norm [162] is employed due to its superiority in training speed and performance [163].”
Do Transformers Really Perform Bad for Graph Representation? (2021, methodology): “This modification has been unanimously adopted by all current Transformer implementations because it leads to more effective optimization [40].”
Do Transformers Really Perform Badly for Graph Representation? (2021, methodology): “This modification has been unanimously adopted by all current Transformer implementations because it leads to more effective optimization [39].”
Designing Effective Sparse Expert Models (2022, result): “This quality gain was corroborated in later work (Narang et al., 2021).”
cosFormer: Rethinking Softmax in Attention (2022, background): “Further, the kernel based methods like (Narang et al., 2021; Choromanski et al., 2020; Tay et al., 2020a) have comparable speed and memory overheads, but their performances are less satisfactory compared to COSFORMER across above metrics.”
Unifying Language Learning Paradigms (2022, result): “This is also in light of results such as (Narang et al., 2021).”