An Introductory Survey on Attention Mechanisms in NLP Problems

By Dichao Hu. 2018

276 citations recorded by the source corpus.

First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search (2021, background): “Inthe ﬁeld of Natural Language Processing (NLP), unprecedented results have been achieved by trans-formers architectures (Hu, 2019).”
Transformers in Vision: A Survey (2021, background): “Other literature reviews mainly focus on the NLP domain [85, 194] or cover generic attentionbased approaches [25, 85].”
A General Survey on Attention Mechanisms in Deep Learning (2022, background): “For example, in [18], a survey is presented on attention in computer vision, [19] provides an overview of attention in graph models, and [20], [21], [22] are all surveys on attention in natural language processing.”
Earthquake transformer—an attentive deep-learning model for simultaneous earthquake detection and phase picking (2020, methodology): “Attention mechanism is a method of encoding sequence data in which elements within a sequence will be highlighted or down-weighted based on their importance or irrelevance to a task 32 – 35 .”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, background): “The attention mechanism (AM) in deep learning mimics the way humans think and is widely used in various types of deep learning tasks such as Natural Language Processing (NLP), image classiﬁcation, and speech recognition [110]–[112].”
Natural language processing applied to mental illness detection: a narrative review (2022, background): “Other emerging techniques like attention mechanism 187 , knowledge graph 188 , and commonsense reasoning 189 , can also be introduced for textual feature extraction.”
Explainability for Large Language Models: A Survey (2023, background): “Firstly, as models become larger, understanding and interpreting their decision-making processes grows more difficult due to increased internal complexity and the vastness of training data.”
Attention, please! A survey of neural attention models in deep learning (2021, background): “Wang and Tax (2016) propose a review on recurrent networks and applications in computer vision, Hu (2019), and Galassi et al.”
EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention (2023, background): “Recently, deep learning has been demonstrated to outperform traditional machine learning in many ﬁelds, e.g., computer vision [19], natural language processing [20] and biomedical signal processing [21], [22], [23].”