DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding

By Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, Chengqi Zhang. 2017

787 citations recorded by the source corpus.

Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)," is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02% on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Pay Less Attention with Lightweight and Dynamic Convolutions (2019, background): “Shen et al. (2018b) reduce complexity by performing attention within blocks of the input sequence and Shen et al. (2017; 2018c) perform more ﬁne-grained attention over each feature.”
Are Sixteen Heads Really Better than One? (2019, background): “Since then, several extensions to the general methodology have been proposed (Ahmed et al., 2017; Shen et al., 2018).”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “For the computational complexity, our model is almost same order as the most simple SWEM model, and is smaller than the CNN or LSTM by a factor of mh/K or h/K .”
Modeling Intra-Relation in Math Word Problems with Different Functional Multi-Head Attentions (2019, background): “…networks have shown impressive results in various natural language processing tasks, such as machine translation (Vaswani et al., 2017; Shaw et al., 2018) and natural language inference (Shen et al., 2018) due to their flexibility in parallel computation and power of modeling long dependencies.”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, methodology): “token2token self-attention (Hu et al., 2017; Vaswani et al., 2017; Shen et al., 2017) produces context-aware representations by exploring the dependency between two tokens x i and x j from the same sequence x .”
Deep Learning for Entity Matching: A Design Space Exploration (2018, background): “A different line of work [59, 65] suggested that using only attention mechanisms with simple feed-forward NNs suffice to summarize pairs of sentences, thus avoiding learning complicated representations such as those obtained by RNNs.”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “The length generalization problem has been reported even in pretrained Transformers such as T5 (Furrer et al., 2020) and LaMDA (Anil et al., 2022).”
Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, positive): “The results presented here thus theoretically confirm the intuition that models entirely built on self-attention may have restricted expressivity when compared to recurrent architectures (Tran et al., 2018; Dehghani et al., 2019; Shen et al., 2018a; Chen et al., 2018; Hao et al., 2019).”
Convolutional Interaction Network for Natural Language Inference (2018, methodology): “The role of the intra-sentence convolutional interaction is the same as self-attention Shen et al. (2017), which is also very useful in NLI.”
Message Passing Attention Networks for Document Understanding (2019, methodology): “DiSAN (Shen et al. 2018) uses directional self-attention along with multi-dimensional attention to generate document representations.”
Deep Learning--based Text Classification (2020, background): “[81] present a directional self-attention network for RNN/CNN-free language understanding, where the attention between elements from input sequence(s) is directional and multidimensional.”
DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks (2020, methodology): “Another potential design choice is applying self-attention along the two dimensions of neighbors and time together following a strategy similar to DiSAN [28].”
A Compare-Propagate Architecture with Alignment Factorization for Natural Language Inference (2017, methodology): “For instance, models that leverage directional selfattention (Shen et al., 2017) or Gumbel-Softmax (Choi et al.”
EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention (2023, methodology): “For example, Vaswani et al. proposed an attention-based architecture for machine translation [32], and Shen et al. proposed the directional self-attention network to focus on the attention among elements in an input sequence [29].”
Distance-based Self-Attention Network for Natural Language Inference (2017, methodology): “BiLSTM encoders (Chen et al., 2017) 11.6m 90.5 85.5 600D Directional Self-Attention Network (Shen et al., 2017) 2 The improvement of the test accuracy by introducing the distance mask is only by 0.3% point, potentially because SNLI data mostly consist of short sentences.”