Blockwise Self-Attention for Long Document Understanding

By Jiezhong Qiu, Hao Ma, Omer Levy, Scott Yih, Sinong Wang, Jie Tang. 2019

280 citations recorded by the source corpus.

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Big Bird: Transformers for Longer Sequences (2020, methodology): “Ye et al. [104] proposed binary partitions of the data where as Qiu et al. [73] reduced complexity by using block sparsity.”
Self-attention Does Not Need $O(n^2)$ Memory (2021, background): “…and architectures with more favorable complexity classes (Kitaev et al., 2020; Roy et al., 2021; Zaheer et al., 2020; Choromanski et al., 2020; Wang et al., 2020; Ren et al., 2021; Child et al., 2019; Tay et al., 2021; Wang et al., 2020; Ma et al., 2021; Shen et al., 2021; Qiu et al., 2020).”
Longformer: The Long-Document Transformer (2020, methodology): “Blockwise attention (Qiu et al., 2019) pretrained their models and evaluated on question answering (QA).”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “Most works (Liu et al., 2018a; Parmar et al., 2018; Child et al., 2019; Huang et al., 2019; Ho et al., 2019; Hu et al., 2019; Parmar et al., 2019; Qiu et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) follow this paradigm.”
Efficient Transformers: A Survey (2020, methodology): “Examples of models that do this include Blockwise (Qiu et al., 2019) and/or Local Attention (Parmar et al., 2018).”
Linformer: Self-Attention with Linear Complexity (2020, methodology): “One popular technique is introducing sparsity into attention layers (Child et al., 2019; Qiu et al., 2019; Beltagy et al., 2020) by having each token attend to only a subset of tokens in the whole sequence.”
Sparse Sinkhorn Attention (2020, background): “been an emerging field of research (Shen et al., 2018b; Tay et al., 2019; Qiu et al., 2020; Child et al., 2019; Parmar et al., 2018).”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, background): “The first is the sparse attention mechanism, including contentindependent sparsity [28, 8, 29, 14] and content-dependent sparsity [17, 32, 37, 51].”
Random Feature Attention (2021, background): “The sparse patterns can be set a priori (Liu et al., 2018; Qiu et al., 2020; Ho et al., 2020; You et al., 2020, inter alia ) or learned from data (Sukhbaatar et al., 2019; Roy et al., 2020, inter alia ).”
FNet: Mixing Tokens with Fourier Transforms (2021, background): “(2020d) survey many of the recent efficient attention works, which they broadly categorize as: (1) Data independent sparsity approaches that use fixed attention patterns (Child et al., 2019; Qiu et al., 2019; Parmar et al., 2018; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020); (2) ”
XCiT: Cross-Covariance Image Transformers (2021, background): “These include restricting the span of the self-attention to local windows [48, 50], strided patterns [14], axial patterns [30], or an adaptive computation across layers [57].”
Luna: Linear Unified Nested Attention (2021, methodology): “Popular methods include local attention (Parmar et al., 2018), blockwise attention (Qiu et al., 2019), strided attention patterns (Child et al., 2019; Beltagy et al., 2020), and compressed attention (Liu et al., 2018).”
GMAT: Global Memory Augmentation for Transformers (2020, methodology): “To alleviate this issue, several sparsifications of vanilla self-attention have been recently proposed; each restricting the number of positions that a given position can attend to [28, 2, 21, 30, 20, 9, 1].”
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021, unknown): “And some works attempt to reduce the computational complexity with sliding window based self-attention (Beltagy et al., 2020; Qiu et al., 2020; Zaheer et al., 2020).”
Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Document Matching (2020, background): “Recently there are some related works on adapting Trans-formers for long text modeling [1, 5, 10, 15, 21, 25, 27, 29, 36, 40].”
Diagonal State Spaces are as Effective as Structured State Spaces (2022, unknown): “To alleviate this issue, several Transformer variants have been proposed with reduced compute and memory requirements (Qiu et al., 2020; Kitaev et al., 2020; Roy et al., 2021; Beltagy et al., 2020; Gupta and Berant, 2020; Katharopoulos et al., 2020; Wang et al., 2020; Choromanski et al., 2021; Zhu a”
Long Range Language Modeling via Gated State Spaces (2022, background): “, 2020), and Blockwise self-attention (Qiu et al., 2020), Reformer (Kitaev et al.”
A Practical Survey on Faster and Lighter Transformers (2021, methodology): “[77] introduced BlockBERT, which relies on the block-wise attention: the input sequence is split into nb non-overlapping blocks, and positions in block i are only allowed to attend to positions in block π(i), where π denotes a permutation.”