An Attentive Survey of Attention Models

By Sneha Chaudhari, Gungor Polatkan, Rohan Ramanath, Varun Mithal. 2019

766 citations recorded by the source corpus.

Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy that groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Evaluation & Benchmarks

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Readers who are interested in this topic may refer to review studies [139], [140], [141].”
This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition (2020, background): “This issue may also arise with other explainability methods that show or high-light image parts, such as attention mechanisms [10, 13, 44], components [37] and other part-based models e.g. [43, 45, 46].”
A Survey of Deep Learning-Based Object Detection (2019, background): “Some typical works [73]–[79] focus on attention mechanism so as to capture more useful features what detecting objects need.”
Attention mechanism in neural networks: where it comes and where it goes (2022, background): “Previous work reviewed the attention mechanisms from different perspectives [15], or examined them in context of natural language processing (NLP) [16, 17].”
Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions (2021, background): “Similarly, domain knowledge in medical image analysis [58], financial sentiment analysis [49], cybersecurity analytics [94, 103] as well as conceptual data model in which semantic information, (i.e., meaningful for a system, rather than merely correlational) [45, 121, 131] is etc [13, 80].”
A General Survey on Attention Mechanisms in Deep Learning (2022, background): “For example, in [18], a survey is presented on attention in computer vision, [19] provides an overview of attention in graph models, and [20], [21], [22] are all surveys on attention in natural language processing.”
AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks (2021, background): “Attention mechanism was ﬁrst introduced in machine translation task [34] and has now become an important concept in artiﬁcial intelligence, because it can improve the interpretability of neural networks, which are otherwise considered as black-box models [35], [36].”
Attention mechanisms in computer vision: A survey (2021, background): “[141] provided a survey of attention models in deep neural networks which concentrates on their application to natural language processing, while our work focuses on computer vision.”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “In recent years, the attention mechanism (AM) has become one of the most challenging tasks in computer vision and machine translation [232].”
Obtaining genetics insights from deep learning via explainable artificial intelligence (2022, background): “The most widely used architectures for sequence- to-activity models are convolutional neural networks (CNNs) and recurrent neural networks (RNNs) 2 , both of which were initially developed for tasks in computer vision and natural language processing (see Box 2 for an overview of neural networks).”