Attention in Natural Language Processing

By Andrea Galassi, Marco Lippi, Paolo Torroni. IEEE Transactions on Neural Networks and Learning Systems, 2019

585 citations recorded by the source corpus.

Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ab456c1ed181c5c48a34adb61395d4806a0ba949

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Classification of Remote Sensing Images Using EfficientNet-B3 CNN Model With Attention (2021, background): “The attention mechanism (the idea of focusing on spe-ciﬁc parts of the input) has been applied in deep learning for speech recognition [30], Natural Language processing [31], multimodal reasoning and matching [32], object detection [33], and image recognition [34]–[36].”
A Survey of Visual Transformers (2021, background): “Inspired by the great success of such self-attention mechanisms for the NLP tasks [1], [13], some CNN-based models attempted to capture the long-range dependencies through adding a self-attention layer at either spatial level [14]–[16] or channel level [17]–[19], while others try to replace the trad”
An Attentive Survey of Attention Models (2019, methodology): “2019], and Natural Language Processing [Galassi et al. 2020].”
A Practical Survey on Faster and Lighter Transformers (2021, methodology): “idden representations h i for i= 1;:::;nwhere the weights are computed with a feed-forward neural network. For a comprehensive survey of the attention mechanism, we refer the reader to Galassi et al. [7] and Weng [8]. Figure 3 illustrates the inter-attention framework. Moreover, recurrent neural net”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Different from the similar studies, BIGBIRD performs well for genomics data alongside NLP tasks such as question answering.”