Neural Machine Translation with Recurrent Attention Modeling

By Chris Dyer, Zhiting Hu, Alex Smola, Zichao Yang, Yuntian Deng. 2016

53 citations recorded by the source corpus.

Knowing which words have been attended to in previous time steps while generating a translation is a rich source of information for predicting what words will be attended to in the future. We improve upon the attention model of Bahdanau et al. (2014) by explicitly modeling the relationship between previous and subsequent attention levels for each word using one recurrent network per input word. This architecture easily captures informative features, such as fertility and regularities in relative distortion. In experiments, we show our parameterization of attention improves translation quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “Overall, our attention mechanism considers which words we previously attended to (Yang et al., 2016) and performs multiple attention ’hops’ per time step.”
Neural Machine Translation With GRU-Gated Attention Model (2020, background): “Mi et al. [13] leverage well-trained word alignments to directly supervise the attention weights in NMT. Yang et al. [14] bring in a recurrence along the context vectors to help adjust future attention.”
Document-Level Neural Machine Translation with Hierarchical Attention Networks (2018, background): “Sentence-level NMT Initial studies on NMT enhanced the sentence-level context by using memory networks (Wang et al., 2016), self-attention (Miculicich Werlen et al., 2018; Zhang et al., 2016), and latent variables (Yang et al., 2017).”
An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation (2016, methodology): “As the score function we use in this paper has relatively low computation cost, the difference of real decoding speed is expected to be enlarged with more complicated attention models, such as Recurrent Attention (Yang et al., 2016) and Neural Tensor Network (Socher et al., 2013).”
Accelerating Neural Transformer via an Average Attention Network (2018, background): “Yang et al. (2017) introduce a recurrent cycle on the attention layer to enhance the model’s memorization of previous translated source words.”
Neural Machine Translation: A Review (2019, background): “Alternatively, researchers have tried to equip the attention layer itself with additional components like a memory [404] or a recurrent network [405, 406] to enable it to keep track of the attention history.”
Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue (2020, methodology): “Another related class of sequential latent models may be latent attention models (Deng et al., 2018; Wang et al., 2018; Yang et al., 2017), which exploit the latent variables to model the attention mapping between input and output sequences.”