Big Bird: Transformers for Longer Sequences

By Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, J. Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang and 1 more. 2020

2,963 citations recorded by the source corpus.

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

Subfields: Artificial Intelligence, Molecular Biology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Jianfeng Gao
3. Luke S. Zettlemoyer
4. Quoc V. Le
5. Percy Liang
6. Graham Neubig
7. Caiming Xiong
8. Noah A. Smith

In Molecular Biology

1. Michael Heinzinger
2. Quoc V. Le
3. Jure Leskovec
4. Frank Hutter
5. Stefano Ermon
6. Zhiyuan Liu
7. Christian Dallago
8. Xiaodan Liang

In Model Architectures

1. Quoc V. Le
2. Jianfeng Gao
3. Yoshua Bengio
4. Alan L. Yuille
5. Song Han
6. Yi Tay
7. Barret Zoph
8. Noah A. Smith

Citations in the corpus

• PaLM: Scaling Language Modeling with Pathways (2022, background): “Sparsity in the sequence length is an area that allows for training efficiently with extremely long sequences (Zaheer et al., 2020; Tay et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Roy et al., 2020; Child et al., 2019; Gale et al., 2019).”
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, methodology): “√ For large sequence lengths 𝑁 , 𝑠 is often set to 𝑁 − 1 / 2 [12] or 𝑁 − 1 log 𝑁 [3, 18, 96], resulting in Θ ( 𝑁 𝑁 ) or Θ ( 𝑁 log 𝑁 ) IO complexity.”
• Self-attention Does Not Need $O(n^2)$ Memory (2021, background): “So the space complexity of transformers is a particular concern, c.f. Kitaev et al. (2020); Roy et al. (2021); Zaheer et al. (2020).”
• BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “Natural language generation models are the central building blocks for many important artificial intelligence applications, including machine translation [17], text summarization [44], automatic storytelling [43], conversation bots [20], and writing assistants [39].”
• Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “Finally, we point out the aforementioned data structures can also be applied on top of our method to search for relevant blocks.”
• ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “It uses a fine-tuned document level BigBird model (Zaheer et al., 2020) for this purpose.”
• Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “include XLNet [Yang et al., 2019f], RoBERTa [Liu et al., 2019c], ALBERT [Lan et al., 2020], Megatron-LM [Shoeybi et al., 2019], ELECTRA [Clark et al., 2020], Reformer [Kitaev et al., 2020], Big Bird [Zaheer et al., 2020], and many more. Although a major focus of this survey is BERT, many of the same”
• CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “The pivot tokens are image tokens selected at random, similar to big bird [53].”
• Measuring Attribution in Natural Language Generation Models (2021, methodology): “We also used an abstractive summarization system, BigBird (Zaheer et al., 2020).”
• Survey of Hallucination in Natural Language Generation (2022, background): “[59], it would be beneficial to apply other works which are aiming to grasp the longer context but do not focus on dialogue generation [8, 223, 233].”
• UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “We observe that the accuracy increases as the input becomes longer, motivating future work to study how to effectively encode large structured input, e.g., leveraging sparse attention (Zaheer et al., 2020). point selection.”
• Sparse is Enough in Scaling Transformers (2021, neutral): “Several recent papers studied this dataset and tasks and it has been shown [46, 45] that pretraining on C4 yields significant improvements on this task.”
• Scalable Adaptive Computation for Iterative Generation (2022, background): “…explored more dynamic and input-decoupled computation, e.g., networks with auxiliary memory (Dai et al., 2019; Rae et al., 2019) and global units (Zaheer et al., 2020; Burtsev et al., 2020; Jaegle et al., 2021b;a), general architectures that leverage adaptive computation to effectively scale to…”
• Generating Images with Sparse Representations (2021, background): “While memory-efficient Transformer variants (Child et al., 2019; Kitaev et al., 2020; Choromanski et al., 2021; Dhari-wal et al., 2020; Zaheer et al., 2021; Roy et al., 2020) make long-sequence training more practical, care is still required to optimize model hyperparameters against available memory…”
• Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “Beltagy et al. (2020); Ainslie et al. (2020); Zaheer et al. (2020) allow a small number of special tokens having access to all key elements.”
• Long Range Arena: A Benchmark for Efficient Transformers (2020, methodology): “Thirdly, many papers tend to conflate the effectiveness of the inductive bias with the benefits of pretraining (Ainslie et al., 2020; Preprint Zaheer et al., 2020; Wang et al., 2020), which tends to obfuscate the true value of the architecture.”
• Efficient Transformers: A Survey (2020, methodology): “The BigBird model does not introduce new parameters beyond the Transformer model.”