Structured Attention Networks

By Yoon Kim, Carl Denton, Luong Hoang, Alexander M. Rush. 2017

496 citations recorded by the source corpus.

Attention networks have proven to be an effective approach for embedding categorical inference within a deep neural network. However, for many tasks we may want to model richer structural dependencies without abandoning end-to-end training. In this work, we experiment with incorporating richer structural distributions, encoded using graphical models, within deep networks. We show that these structured attention networks are simple extensions of the basic attention procedure, and that they allow for extending attention beyond the standard soft-selection approach, such as attending to partial segmentations or to subtrees. We experiment with two different classes of structured attention networks: a linear-chain conditional random field and a graph-based parsing model, and describe how these models can be practically implemented as neural network layers. Experiments show that this approach is effective for incorporating structural biases, and structured attention networks outperform baseline attention models on a variety of synthetic and real tasks: tree transduction, neural machine translation, question answering, and natural language inference. We further find that models trained in this way learn interesting unsupervised hidden representations that generalize simple attention.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Attention is All you Need (2017, background): “Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences [2, 16].”
Attention is not Explanation (2019, background): “Structured attention models (Kim et al., 2017) provide a generalized framework for describing and fitting attention variants with explicit probabilistic semantics.”
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points (2018, neutral): “Soft attention weights each part of the observation dynamically [4, 25].”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, methodology): “Soft attention was used for various applications such as neural machine translation [5, 17] or image captioning [41].”
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering (2018, methodology): “Structured attention [21] adopted the MRF model over attention maps for better modelling better spatial attention distributions.”
Weighted Transformer Network for Machine Translation (2017, background): “Attention mechanisms, often used in conjunction with recurrent models, have become an integral part of complex sequential tasks because they facilitate learning of such dependencies (Luong et al., 2015; Bahdanau et al., 2014; Parikh et al., 2016; Paulus et al., 2017; Kim et al., 2017).”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, background): “(Kim et al., 2017) also make the connection between soft attention and selecting items from memory in expectation.”
Learning to Count Objects in Natural Images for Visual Question Answering (2018, background): “Hard attention (Ba et al., 2015; Mnih et al., 2014) and structured attention (Kim et al., 2017) may be possible solutions to this, though no signiﬁcant improvement in counting ability has been found for the latter so far (Zhu et al., 2017).”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, methodology): “To obtain long-term dependency context-aware sequence representation, structured attention (Kim et al. 2017; Liu and Lapata 2017) and self-alignment attention (Hu, Peng, and Qiu 2017) are proposed, using cosine similarity between each pair of element form input sequence to mitigate the weakness of…”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “However, only soft attention is used in the majority of Visual QA works [7,8,9,10,11,12,43,44,45,46,47,48,49,50,51,52].”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, background): “Recently, several structured attention mechanisms (Kim et al., 2017; Kokkinos & Potamianos, 2017) are proposed for capturing structural information from input sequence(s).”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, background): “Other works also consider modifications to the attention mechanism in order to improve NMT (Kim et al., 2017).”
A Bichannel Transformer with Context Encoding for Document-Driven Conversation Generation in Social Media (2020, background): “Attention mechanisms have become an integral part of sequence models in response generation, modeling the textual dependencies in the input or output sequences without regard to the position information [35, 36].”
Reasoning with Latent Structure Refinement for Document-Level Relation Extraction (2020, methodology): “The structure induction module is built based on the structured attention (Kim et al., 2017; Liu and Lapata, 2018).”
Semi-Amortized Variational Autoencoders (2018, background): “, 2015) and as a differentiable layer of a deep model (Belanger et al., 2017; Kim et al., 2017; Metz et al., 2017; Amos & Kolter, 2017).”
Attention in Natural Language Processing (2019, background): “For example, structured attention networks [47] exploit neural conditional random fields to model the (conditional) attention distribution.”
Unsupervised Neural Single-Document Summarization of Reviews via Learning Latent Discourse Structure and its Ranking (2019, methodology): “Inspired by structure bias (Cheng and Lapata, 2016; Kim et al., 2017), they introduced Structured Attention, which normalizes attention scores as the posterior marginal probabilities of a nonprojective discourse tree.”
Latent Alignment and Variational Attention (2018, background): “Recent work has combined neural attention with traditional alignment [18, 72] and induced structure/sparsity [48, 33, 44, 85, 54, 55, 49], which can be combined with the variational approaches outlined in this paper.”