Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling

By Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, Chengqi Zhang. 2018

148 citations recorded by the source corpus.

Many natural language processing tasks solely rely on sparse dependencies between a few tokens in a sentence. Soft attention mechanisms show promising performance in modeling local/global dependencies by soft probabilities between every two tokens, but they are not effective and efficient when applied to long sentences. By contrast, hard attention mechanisms directly select a subset of tokens but are difficult and inefficient to train due to their combinatorial nature. In this paper, we integrate both soft and hard attention into one context fusion model, "reinforced self-attention (ReSA)", for the mutual benefit of each other. In ReSA, a hard attention trims a sequence for a soft self-attention to process, while the soft attention feeds reward signals back to facilitate the training of the hard one. For this purpose, we develop a novel hard attention called "reinforced sequence sampling (RSS)", selecting tokens in parallel and trained via policy gradient. Using two RSS modules, ReSA efficiently extracts the sparse dependencies between each pair of selected tokens. We finally propose an RNN/CNN-free sentence-encoding model, "reinforced self-attention network (ReSAN)", solely based on ReSA. It achieves state-of-the-art performance on both the Stanford Natural Language Inference (SNLI) and the Sentences Involving Compositional Knowledge (SICK) datasets.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017), but recent non-attentional models designed to extract useful sentence representations have nearly closed the gap (Liu et al., 2017b; Im and Cho, 2017; Shen et al., 2018; Choi et al., 2017).”
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (2018, methodology): “Sentence-encoding models (e.g. Bowman et al., 2015, 2016; Nie and Bansal, 2017; Shen et al., 2018) encode the premise and hypothesis individually, while attention-based models align words in the premise with similar words in the hypothesis, encoding the two sentences together (e.”
Sparse Sinkhorn Attention (2020, background): “The key idea behind sparse weights (i.e., hard attention) is that they enable the model to only focus on a limited number of items at a time (Xu et al., 2015; Shen et al., 2018a).”
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information (2018, methodology): “The first type of these approaches is sentence-encoding-based methods (Conneau et al. 2017; Choi, Yoo, and goo Lee 2017; Nie and Bansal 2017; Shen et al. 2018) where sentences are encoded into their own sentence representation without any cross-interaction.”
Look Harder: A Neural Machine Translation Model with Hard Attention (2019, background): “Even though RL based models are difﬁcult to train, in recent years, multiple works (Mnih et al., 2014; Choi et al., 2017; Yu et al., 2017; Narayan et al., 2018; Sathish et al., 2018; Shen et al., 2018) have shown to improve the performance of several nat-ural language processing tasks.”
Causal Discovery with Attention-Based Convolutional Neural Networks (2019, background): “However, hard attention is non-differentiable due to its discrete nature, and therefore cannot be optimized through backpropagation [50].”
Star-Transformer (2019, methodology): “…300D DiSAN (Shen et al., 2018a) 85.6 Residual encoders (Nie and Bansal, 2017) 86.0 Gumbel TreeLSTM (Choi et al., 2018) 86.0 Reinforced self-att (Shen et al., 2018b) 86.3 2400D Multiple DSA (Yoon et al., 2018) 87.4 The SNLI dataset is not a small dataset in NLP area, so improving the…”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, methodology): “For BiLSTM-Max, our contextual gating sets a new state-of-the-art on SNLI sentence encoding meth-ods (methods without interaction layers), outper-forming the recently proposed methods of Im and Cho (2017); Shen et al. (2018).”
Attention in Natural Language Processing (2019, background): “[94] combined soft and hard attention, by applying the former only on the elements filtered by the latter.”
Neural Natural Language Inference Models Enhanced with External Knowledge (2017, background): “…al., 2015), CNN (Mou et al., 2016), BiL-STM and its variants (Liu et al., 2016c; Lin et al., 2017; Chen et al., 2017b; Nie and Bansal, 2017), self-attention network (Shen et al., 2017, 2018), and more complicated neural networks (Bowman et al., 2016; Yu and Munkhdalai, 2017a,b; Choi et al., 2017).”
A General Survey on Attention Mechanisms in Deep Learning (2022, methodology): “Another proposed method for compromising between soft and hard alignment is reinforced alignment [58].”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “ReSA selects a subset of head tokens, and relates each head token to a small subset of dependent tokens to generate their context-aware representations [105].”
Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, methodology): “The results presented here thus theoretically conﬁrm the intuition that models entirely built on self-attention may have restricted expressivity when compared to recurrent architectures (Tran et al., 2018; Dehghani et al., 2019; Shen et al., 2018a; Chen et al., 2018; Hao et al., 2019).”
Enhancing Sentence Embedding with Generalized Pooling (2018, methodology): “The previous state-of-the-art model ReSAN (Shen et al., 2018) used a hybrid of hard and soft attention model with reinforced learning achieved an accuracy of 86.3%.”