A Structured Self-attentive Sentence Embedding

By Zhouhan Lin, Minwei Feng, C. D. Santos, Mo Yu, Bing Xiang, Bowen Zhou, Yoshua Bengio. 2017

2,310 citations recorded by the source corpus.

This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Attention is All you Need (2017, background): “Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 22, 23, 19].”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “Following Lin et al. (2017) we use a self-attentive network with multiple views of the input sentence, so that the model can learn which part of the sentence is important for the given task.”
Graph Attention Networks (2017, background): “Together with Recurrent Neural Networks (RNNs) or convolutions, self-attention has proven to be useful for tasks such as machine reading (Cheng et al., 2016) and learning sentence representations (Lin et al., 2017).”
Universal Transformers (2018, methodology): “The Transformer model in particular relies entirely on a self-attention mechanism (Parikh et al., 2016; Lin et al., 2017) to compute a series of context-informed vector-space representations of the symbols in its input and output, which are then used to predict distributions over subsequent symbols…”
Object-Centric Learning with Slot Attention (2020, background): “Neural networks for sets A range of recent methods explore set encoding [34, 50, 51], generation [31, 52], and set-to-set mappings [20, 53].”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, neutral): “Transformer network (Vaswani et al., 2017) was ﬁrst introduced for neural machine translation (NMT) tasks, where the encoder and decoder side each leverages a self-attention (Parikh et al., 2016; Lin et al., 2017; Vaswani et al., 2017) transformer.”
Stochastic Answer Networks for Machine Reading Comprehension (2017, methodology): “Inspired by (Lin et al., 2017), we apply a self-attended layer to rearrange the information U p as: (3) In other words, we ﬁrst obtain an n × n attention matrix with U p onto itself, apply dropout, then multiply this matrix with U p to obtain an updated ˆ U p .”
End-to-End Dense Video Captioning with Masked Transformer (2018, neutral): “Self-attention [22, 25, 30] allows for an attention mechanism within a module and is a potential way to learn this long-range dependence.”
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent (2017, methodology): “For image classiﬁcation we train ResNet [He et al., 2015] with different number of layers on CIFAR-10 [Krizhevsky, 2009]; for speech recognition, we train both proprietary and public dataset on a proprietary CNN model that we get from our industry partner [Feng et al., 2016, Lin et al., 2017].”
Relational inductive biases, deep learning, and graph networks (2018, background): “(2018c)’s NLNN, which unifies various “intra-/self-/vertex-/graph-attention” approaches (Lin et al., 2017; Vaswani et al., 2017; Hoshen, 2017; Veličković et al., 2018; Shaw et al., 2018), can also be translated into the GN formalism.”
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, background): “Attention has previously been introduced on graphs under the names of intra-, self-, vertex-, or graph-attention [17, 31, 32, 12, 26].”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, methodology): “Transformer models [6], [77], [78] are ﬁrst proposed to handle natural language processing tasks, e.g., text classi-ﬁcation [79], [80], question answering [81] and machine translation [82].”
Class-Incremental Learning for Action Recognition in Videos (2021, positive): “To further improve the effectiveness of the proposed knowledge distillation strategy, we adopt an additional regularization term inspired by [22], which enforces individual features extracted from different time steps in a video to be mutually independent.”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, methodology): “To better encode a sentence, we use the self-attention layer (Lin et al., 2017) to capture information from critical words.”
Towards Open Intent Discovery for Conversational Text (2019, methodology): “In particular, we use a multi-head self-attention mechanism [20, 30, 44, 46] that jointly attends to information at different positions of the input sequence, with multiple individual attention functions and separately normalized parameters called attention heads.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “, 2014), natural language processing(Vaswani et al., 2017; Bahdanau et al., 2015; Lin et al., 2017), and reinforcement learning (Oh et al.”
Is Attention Interpretable? (2019, methodology): “Since attention calculates a distribution over inputs, prior work has used attention as a tool for interpretation of model decisions (Wang et al., 2016; Lee et al., 2017; Lin et al., 2017; Ghaeini et al., 2018).”
Weighted Transformer Network for Machine Translation (2017, background): “Self-attention, which relies on dot-products between elements of the input sequence to compute a weighted sum (Lin et al., 2017; Bahdanau et al., 2014; Parikh et al., 2016; Kim et al., 2017), has also been a critical ingredient in modern NMT architectures.”