Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling

By Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang. 2018

153 citations recorded by the source corpus.

Recurrent neural networks (RNN), convolutional neural networks (CNN) and self-attention networks (SAN) are commonly used to produce context-aware representations. RNN can capture long-range dependency but is hard to parallelize and not time-efficient. CNN focuses on local dependency but does not perform well on some tasks. SAN can model both such dependencies via highly parallelizable computation, but memory requirement grows rapidly in line with sequence length. In this paper, we propose a model, called "bi-directional block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding. It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN splits the entire sequence into blocks, and applies an intra-block SAN to each block for modeling local context, then applies an inter-block SAN to the outputs for all blocks to capture long-range dependency. Thus, each SAN only needs to process a short sequence, and only a small amount of memory is required. Additionally, we use feature-level attention to handle the variation of contexts around the same word, and use forward/backward masks to encode temporal order information. On nine benchmark datasets for different NLP tasks, Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better efficiency-memory trade-off than existing RNN/CNN/SAN.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “An important next step is thus to speed up the training and inference speed of ALBERT through methods like sparse attention (Child et al., 2019) and block attention (Shen et al., 2018).”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “Shen et al. (2018a) and Gong et al. (2018) use input-dependent ﬁlters for text classiﬁcation tasks.”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “We compare with a variety of methods, including (i) the bag-of-words in (Zhang et al., 2015); (ii) sophisticated deep CNN/RNN models: large/small word CNN, LSTM reported in (Zhang et al., 2015; Dai and Le, 2015) and deep CNN (29 layer) (Conneau et al., 2017); (iii) simple compositional methods: fast”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “…of mechanisms to scale attention to long inputs, including our proposed model, ETC. is not specific to Transformer models, and it has been applied to recurrent neural network models both at the level of sentences (Yang et al., 2016; Miculicich et al., 2018) and blocks (Shen et al., 2018).”
Sparse Sinkhorn Attention (2020, background): “The key idea behind sparse weights (i.e., hard attention) is that they enable the model to only focus on a limited number of items at a time (Xu et al., 2015; Shen et al., 2018a).”
ETC: Encoding Long and Structured Data in Transformers (2020, methodology): “Notice that this idea of processing the input hierarchically is not specific to Transformer models, and it has been applied to recurrent neural network models both at the level of sentences (Yang et al., 2016; Miculicich et al., 2018) and blocks (Shen et al., 2018).”
Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks (2018, neutral): “SFP should also be effective on different computer vision tasks, such as [ Kang et al. , 2017; Ren et al. , 2015; Dong et al. , 2018; Shen et al. , 2018b; Yang et al. , 2010; Shen et al. , 2018a; Dong et al. , 2017b ] , and we will explore this in future.”
Convolutional Self-Attention Networks (2019, methodology): “Along this direction, Shen et al. (2018a) explicitly used multiple attention heads to model different dependencies of the same word pair, and Strubell et al. (2018) employed different attention heads to capture different linguistic features.”
AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search (2020, background): “…operations as they are effective in NLP tasks (Kim, 2014; Bai et al., 2018), and CNN-based operations have shown inference speed superiority over RNN-based models and self-attention based models (Shen et al., 2018; Chia et al., 2018) due to the fact that they are parallel-friendly operations.”
Bidirectional LSTM with attention mechanism and convolutional layer for text classification (2019, background): “Shen et al. [41] proposed bidirectional block self-attention (Bi-BloSA) for fast and memory-eﬃcient context fusion.”
Describing a Knowledge Base (2018, methodology): “…of facts and generating one sentence only, we introduce a table position self-attention, inspired from structure attention (Lin et al., 2017; Kim et al., 2017; Vaswani et al., 2017; Shen et al., 2018a,b), to capture the dependencies among facts and generate a para-graph to describe all facts.”
A Survey on Text Classification: From Shallow to Deep Learning (2020, methodology): “To capture long-range dependencies, Bi-directional Block Self-Attention Network (Bi-BloSAN) [133] uses an intra-block self-attention network (SAN) to every block split by sequence and an inter-block SAN to the outputs.”
Data Movement Is All You Need: A Case Study on Optimizing Transformers (2020, background): “Sparsity optimizations for attention (Bahdanau et al., 2014; Luong et al., 2015; Shen et al., 2018; Parmar et al., 2018; Tay et al., 2019; Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Tay et al., 2020) reduce memory and compute requirements.”
Fast Directional Self-Attention Mechanism (2018, background): “In the future, there are several interesting works can be done: 1) applying more kinds of positional masks [26] or distance-ware masks [14] to different subspaces, 2) applying the Fast-DiSA as a context fusion model to more NLP tasks for a better state-of-the-art performance, and 3) integrating the ”
Convolutional Self-Attention Network (2018, background): “• Concerning the hard fashion, Shen et al. (2018a) and Yu et al. (2018) suggested to allocate hard local scopes through dividing the input sequence into blocks (“BLOCK”), and stacking CNN and SAN layers (“CNNS”).”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Another mechanism, bidirectional block self-attention (BiBloSA) which is simply a masked block self-attention (mBloSA) with forward and backward masks to encode the temporal order information is presented [107].”
A Survey on Text Classification: From Traditional to Deep Learning (2020, methodology): “MR 2 20 10,662 English [5, 17, 101, 155] [226] SA SST-1 5 18 11,855 English [17, 64] [1, 2][112] [227] SA SST-2 2 19 9,613 English [6, 17, 64] [19, 79] [228] SA MPQA 2 3 10,606 English [17, 60, 120] [229] SA IMDB 2 294 50,000 English [69][108] [6] [79] [85] [230] SA Yelp.”