RealFormer: Transformer Likes Residual Attention

By Ruining He, Anirudh Ravula, Bhargav Kanagal, J. Ainslie. 2020

138 citations recorded by the source corpus.

Transformer is the backbone of modern NLP models. In this paper, we propose RealFormer, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention. Source code and pre-trained checkpoints for RealFormer can be found at https://github.com/google-research/google-research/tree/master/realformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Jie Tang
6. Xipeng Qiu
7. Qiang Liu
8. Sebastian Farquhar

In Computer Vision and Pattern Recognition

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Ming Ding
6. Yarin Gal
7. Luc Van Gool
8. Jai Gupta

In Model Architectures

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Jai Gupta
6. Xipeng Qiu
7. Zhen Qin
8. Sebastian Farquhar

Citations in the corpus

• GLM-130B: An Open Bilingual Pre-trained Model (2022, neutral): “This is observed in (He et al., 2021).”
• OmniNet: Omnidirectional Representations from Transformers (2021, background): “The recent Realformer (residual attention) (He et al., 2020) proposed connecting the attention activations in a residual fashion.”
• Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks (2021, background): “Recently, Realformer [16] reveals that residual connection on attention scores is also helpful.”
• A Survey of Transformers (2021, methodology): “4.5.2, we introduced Realformer [51] and Predictive Attention Transformer [143] that reuses attention distributions from previous block to guide attention of current block.”
• Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, background): “(He et al. 2020) proposes to add an evolved global attention to the attention matrix in each layer for a better residual mechanism.”
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention (2021, methodology): “We use the scaled dot-product mechanism based residual attention: ResidualAttention (Q,K,V,Aprev) = Softmax ( QKT√ dk +Aprev ) V (17) We find that using attention weights Aprev makes training process more stable than using before-softmax attention scores, which is used in RealFormer (He et al., 2020”
• DSTAGNN: Dynamic Spatial-Temporal Aware Graph Neural Network for Traffic Flow Forecasting (2022, methodology): “Using the residual attention idea (He et al., 2020), we directly connect the output from the temporal attention module in each ST block with that in the next ST block, which enhances the connection among the temporal attention in different layers of the ST block.”
• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, methodology): “Recently, however, SotA paraphrase identification approaches have primarily used BERT-based models to classify pairs of sequences into the classes paraphrases and not paraphrases (He et al., 2020b; Tay et al., 2021).”