Latent Alignment and Variational Attention

By Yuntian Deng, Yoon Kim, Justin T Chiu, Demi Guo, Alexander M. Rush. 2018

118 citations recorded by the source corpus.

Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property makes it difficult to compare attention to other alignment approaches, to compose it with probabilistic models, and to perform posterior inference conditioned on observed data. A related latent approach, hard attention, fixes these issues, but is generally harder to train and less accurate. This work considers variational attention networks, alternatives to soft and hard attention for learning latent variable alignment models, with tighter approximation bounds based on amortized variational inference. We further propose methods for reducing the variance of gradients to make these approaches computationally feasible. Experiments show that for machine translation and visual question answering, inefficient exact latent variable models outperform standard neural attention, but these gains go away when using hard attention based training. On the other hand, variational attention retains most of the performance gain but with training speed comparable to neural attention.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fixup Initialization: Residual Learning Without Normalization (2019, methodology): “nstead, Fixup matches or supersedes the state-of-the-art results using Transformer model on both datasets. 7 Published as a conference paper at ICLR 2019 Dataset Model Normalization BLEU IWSLT DE-EN (Deng et al., 2018) 3 33.1 LayerNorm 34:2 Fixup-init 7 34.5 WMT EN-DE (Vaswani et al., 2017) 3 28.4 L”
Adaptively Sparse Transformers (2019, background): “Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability (Malaviya et al., 2018; Deng et al., 2018; Peters ”
GMNN: Graph Markov Neural Networks (2019, methodology): “Besides, we can also use the mean pooling method similar to the soft attention method in Deng et al. (2018).”
Autoregressive Knowledge Distillation through Imitation Learning (2020, methodology): “Following standard practice (Bahdanau et al., 2016; Deng et al., 2018; Wang et al., 2019), we randomly sample 4% of this dataset as the validation set and let the remaining be the training set.”
Understanding Multi-Head Attention in Abstractive Summarization (2019, methodology): “Sparsity has been used to improve the interpretability of single-head attention architectures (Malaviya et al., 2018; Deng et al., 2018; Niculae et al., 2018).”
Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020, methodology): “models applied to tasks like machine translation (Wang et al., 2018; Deng et al., 2018), character transduction (Wu et al.”
Mixture Content Selection for Diverse Sequence Generation (2019, background): “…for diverse generation in several tasks, such as language modeling (Bowman et al., 2016), machine translation (Zhang et al., 2016; Su et al., 2018; Deng et al., 2018; Shankar and Sarawagi, 2019), and conversation modeling (Serban et al., 2017; Wen et al., 2017; Zhao et al., 2017; Park et al.,…”
Context-Gated Convolution (2019, unknown): “[13] 33.”
Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue (2020, background): “Another related class of sequential latent models may be latent attention models (Deng et al., 2018; Wang et al., 2018; Yang et al., 2017), which exploit the latent variables to model the attention mapping between input and output sequences.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “[61] proposed using variational attention mechanism.”
Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction (2018, unknown): “RNNsearch* (Bahdanau et al., 2015) 31.02 1.79 6M 25.92 7M Varational attention (Deng et al., 2018) 33.10 Transformer** (Vaswani et al., 2017) 32.83 3.53 59M 27.68 61M ConvS2S** (MLE) (Gehring et al., 2017b) 32.31 1.35 21M 26.73 22M ConvS2S (MLE+SLE) (Edunov et al., 2018) 32.84 Pervasive Attention…”
Unsupervised Recurrent Neural Network Grammars (2019, methodology): “This control variate worked better than alternatives such as estimates of baselines from an auxiliary network (Mnih and Gregor, 2014; Deng et al., 2018) or a language model (Yin et al.”