Universal Transformers

By Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser. 2018

980 citations recorded by the source corpus.

Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been shown to achieve superior results on some sequence modeling tasks such as machine translation, with the added advantage that they concurrently process all inputs in the sequence, leading to easy parallelization and faster training times. Despite these successes, however, popular feed-forward sequence models like the Transformer fail to generalize in many simple tasks that recurrent models handle with ease, e.g. copying strings or even simple logical inference when the string or formula lengths exceed those observed at training time. We propose the Universal Transformer (UT), a parallel-in-time self-attentive recurrent sequence model which can be cast as a generalization of the Transformer model and which addresses these issues. UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. We also add a dynamic per-position halting mechanism and find that it improves accuracy on several tasks. In contrast to the standard Transformer, under certain assumptions, UTs can be shown to be Turing-complete. Our experiments show that UTs outperform standard Transformers on a wide range of algorithmic and language understanding tasks, including the challenging LAMBADA language modeling task where UTs achieve a new state of the art, and machine translation where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Scaling Laws for Neural Language Models (2020, neutral): “We also compare the performance of standard Transformers to recurrent Transformers [DGV + 18] in Figure 17 in the appendix.”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “(4)We also experiment with parameter sharing (Lan et al., 2020; Dehghani et al., 2019)—tying the weights of the Transformer layers together—and find that it slows convergence (see Appendix C).”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “, 2020), weight sharing (Dehghani et al., 2019; Lan et al., 2020), knowledge distillation (Tang et al.”
Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, methodology): “Several methods for algorithm induction speciﬁcally add adaptive computation time to sequence models (Graves, 2016; Dehghani et al., 2018; Banino et al., 2021).”
Rethinking Attention with Performers (2020, background): “Transformers (Vaswani et al., 2017; Dehghani et al., 2019) are powerful neural network architectures that have become SOTA in several areas of machine learning including natural language processing (NLP) (e.g. speech recognition (Luo et al., 2020)), neural machine translation (NMT) (Chen et al.,…”
Emergent Abilities of Large Language Models (2022, neutral): “Other directions for better computational efﬁciency could involve variable amounts of compute for different inputs (Graves, 2016; Dehghani et al., 2018), using more localized learning strategies than backpropagation through all weights in a neural network (Jaderberg et al., 2017), and augmenting…”
Quantifying Attention Flow in Transformers (2020, background): “…the easiest and most popular approach to interpret a model’s decisions and to gain insights about its internals (Vaswani et al., 2017; Xu et al., 2015; Wang et al., 2016; Lee et al., 2017; Dehghani et al., 2019; Rockt¨aschel et al., 2016; Chen and Ji, 2019; Coenen et al., 2019; Clark et al., 2019).”
Perceiver: General Perception with Iterative Attention (2021, methodology): “We note that weight sharing has been used for similar goals in Transformers (Dehghani et al., 2019; Lan et al., 2020).”
Generative Language Modeling for Automated Theorem Proving (2020, background): “The Universal Transformer [40], a Transformer with tied weights, was also shown to be successful at more algorithmic tasks.”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, methodology): “There have been other recent advances in understanding the theory behind transformers: (Perez et al., 2019; Dehghani et al., 2019) proved Turing universality, (Cordonnier et al.”
Of Non-Linearity and Commutativity in BERT (2021, background): “Remarkably, ALBERT [17] directly shares the weights across the layers of BERT without loss of performance, and Universal Transformers [5] use recurrence over one single transformer layer.”
Dynamic Neural Networks: A Survey (2021, methodology): “In addition, the halting score in [11], [31], [33], [34] could also be viewed as confidence for whether the current feature could be output to the next time step or calculation stage.”
Controlling Computation versus Quality for Neural Sequence Models (2020, methodology): “Universal transformer (Dehghani et al., 2018) addressed the parameter inefficiency problem of Transformers by tying the weights of consecutive layers and utilizing ACT to decide the halting of such recurrence.”
Depth-Adaptive Transformer (2019, positive): “Moreover, Dehghani et al. (2018) ﬁx the number of steps for large-scale machine translation whereas we vary the number of steps to demonstrate substantial improvements in speed at no loss in accuracy.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “Universal Transformers (UT) (Dehghani et al., 2018) because it was claimed to achieve better results than the vanilla Transformer, and the UT has a relatively large number of hyperparameters that we can adjust.”
Efficient Transformers: A Survey (2020, background): “As such, it is only natural that a wealth of research has been dedicated to making fundamental improvements to the model over the past few years (Dehghani et al., 2018; So et al., 2019; Ahmed et al., 2017).”
On Layer Normalization in the Transformer Architecture (2020, background): “This architecture has achieved state-of-the-art performance in many tasks including language modeling (Dai et al., 2019; Al-Rfou et al., 2018) and machine translation (Dehghani et al., 2018; Edunov et al., 2018).”