Distance-based Self-Attention Network for Natural Language Inference

By Jinbae Im, Sungzoon Cho. 2017

76 citations recorded by the source corpus.

Attention mechanism has been used as an ancillary means to help RNN or CNN. However, the Transformer (Vaswani et al., 2017) recently recorded the state-of-the-art performance in machine translation with a dramatic reduction in training time by solely using attention. Motivated by the Transformer, Directional Self Attention Network (Shen et al., 2017), a fully attention-based sentence encoder, was proposed. It showed good performance with various data by using forward and backward directional information in a sentence. But in their study, not considered at all was the distance between words, an important feature when learning the local dependency to help understand the context of input text. We propose Distance-based Self-Attention Network, which considers the word distance by using a simple distance mask in order to model the local dependency without losing the ability of modeling global dependency which attention has inherent. Our model shows good performance with NLI data, and it records the new state-of-the-art result with SNLI data. Additionally, we show that our model has a strength in long sentences or documents.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017), but recent non-attentional models designed to extract useful sentence representations have nearly closed the gap (Liu et al., 2017b; Im and Cho, 2017; Shen et al., 2018; Choi et al., 2017).”
Towards Open Intent Discovery for Conversational Text (2019, methodology): “In particular, we use a multi-head self-attention mechanism [20, 30, 44, 46] that jointly attends to information at different positions of the input sequence, with multiple individual attention functions and separately normalized parameters called attention heads.”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, methodology): “For BiLSTM-Max, our contextual gating sets a new state-of-the-art on SNLI sentence encoding meth-ods (methods without interaction layers), outper-forming the recently proposed methods of Im and Cho (2017); Shen et al. (2018).”
Self-Attentional Acoustic Models (2018, background): “3 Note that this bears some resemblance to prior work [15] who use a linear distance map instead of a Gaussian and do not include trainable parameters, making their model less ﬂexible and less interpretable.”
Fast Directional Self-Attention Mechanism (2018, methodology): “Distance-based SAN is even less time efficient than DiSA as shown in [14]”