GMAT: Global Memory Augmentation for Transformers

By Ankit Gupta, Jonathan Berant. 2020

53 citations recorded by the source corpus.

Transformer-based models have become ubiquitous in natural language processing thanks to their large capacity, innate parallelism and high performance. The contextualizing component of a Transformer block is the $\textit{pairwise dot-product}$ attention that has a large $\Omega(L^2)$ memory requirement for length $L$ sequences, limiting its ability to process long documents. This has been the subject of substantial interest recently, where multiple approximations were proposed to reduce the quadratic memory requirement using sparse attention matrices. In this work, we propose to augment sparse Transformer blocks with a dense attention-based $\textit{global memory}$ of length $M$ ($\ll L$) which provides an aggregate global view of the entire input sequence to each position. Our augmentation has a manageable $O(M\cdot(L+M))$ memory overhead, and can be seamlessly integrated with prior sparse solutions. Moreover, global memory can also be used for sequence compression, by representing a long input sequence with the memory representations only. We empirically show that our method leads to substantial improvement on a range of tasks, including (a) synthetic tasks that require global reasoning, (b) masked language modeling, and (c) reading comprehension.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Hardware and Architecture

Citations in the corpus

MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Later, the complexity of transformer attention has been reduced [49, 90], by introducing local [68] or sparse attention [23], together with a global memory [6, 103, 31, 1].”
Luna: Linear Unified Nested Attention (2021, background): “…(Dehghani et al., 2018), quantization (Shen et al., 2020; Fan et al., 2020), sparse attention (Parmar et al., 2018; Kitaev et al., 2020), side memory module (Lee et al., 2019; Gupta and Berant, 2020; Goyal et al., 2021), and low-rank or compressed context (Wang et al., 2019; Ainslie , 2020).”
Recurrent Memory Transformer (2022, methodology): “Among them are Star-Transformer (Guo et al., 2019), Longformer (Beltagy et al., 2020), GMAT (Gupta and Berant, 2020), Extended Transformer Construction (ETC) (Ainslie et al., 2020) and Big Bird (Zaheer et al., 2020).”
Simplified State Space Layers for Sequence Modeling (2022, methodology): “In particular, many efficient transformer methods have been introduced [11, 27, 29, 5, 21, 59] to address the standard transformer’s quadratic complexity in the sequence length.”
Diagonal State Spaces are as Effective as Structured State Spaces (2022, unknown): “To alleviate this issue, several Transformer variants have been proposed with reduced compute and memory requirements (Qiu et al., 2020; Kitaev et al., 2020; Roy et al., 2021; Beltagy et al., 2020; Gupta and Berant, 2020; Katharopoulos et al., 2020; Wang et al., 2020; Choromanski et al., 2021; Zhu a”
Long Range Language Modeling via Gated State Spaces (2022, background): “For example, BigBird [Zaheer et al., 2020], GMAT [Gupta and Berant, 2020], Longformer [Beltagy et al., 2020], and Blockwise self-attention [Qiu et al., 2020] all choose particular sparsiﬁcation patterns that are intuitively likely to capture "important" interactions, while the Reformer [Kitaev et al”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, background): “Transformer [80] variants [2, 7, 22, 30, 48, 66, 84, 92] have advanced the state-of-the-art in many natural language processing tasks [25, 26, 75] by capturing relations across modalities [3] or in a single context (self-attention) [21, 80].”