Temporal Attention Model for Neural Machine Translation

By Baskaran Sankaran, Haitao Mi, Yaser Al-Onaizan, Abe Ittycheriah. 2016

63 citations recorded by the source corpus.

Attention-based Neural Machine Translation (NMT) models suffer from attention deficiency issues as has been observed in recent research. We propose a novel mechanism to address some of these limitations and improve the NMT attention. Specifically, our approach memorizes the alignments temporally (within each sentence) and modulates the attention with the accumulated temporal memory, as the decoder generates the candidate translation. We compare our approach against the baseline NMT model and two other related approaches that address this issue either explicitly or implicitly. Large-scale experiments on two language pairs show that our approach achieves better and robust gains over the baseline and related NMT approaches. Our model further outperforms strong SMT baselines in some settings even without using ensembles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Caiming Xiong
3. Richard Socher
4. Mirella Lapata
5. Jianfeng Gao
6. Luke S. Zettlemoyer
7. Christopher D. Manning
8. Yejin Choi

In Computer Vision and Pattern Recognition

1. Caiming Xiong
2. Richard Socher
3. Luc Van Gool
4. Trevor Darrell
5. Dacheng Tao
6. Mohit Bansal
7. Xiaogang Wang
8. Cordelia Schmid

In Natural Language Processing

1. Caiming Xiong
2. Mohit Bansal
3. Richard Socher
4. Mirella Lapata
5. Christopher D. Manning
6. Graham Neubig
7. Luke S. Zettlemoyer
8. Furu Wei

In Model Architectures

1. Richard Socher
2. Caiming Xiong
3. Christopher D. Manning
4. Peter J. Liu
5. Abigail See
6. Yang Fang
7. Weidong Xiao
8. Yuheng Wang

Citations in the corpus

• A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “At each decoding step t , we use an intra-temporal attention function to attend over specific parts of the encoded input sequence in addition to the decoder’s own hidden state and the previously-generated word (Sankaran et al., 2016).”
• Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “The pointer network has been used to create hybrid approaches for NMT (Gul-cehre et al., 2016), language modeling (Merity et al., 2016), and summarization (Gu et al., 2016; Gulcehre et al., 2016; Miao and Blunsom, 2016; Nallapati et al., 2016; Zeng et al., 2016).”
• Abstract Text Summarization with a Convolutional Seq2seq Model (2019, methodology): “Reference [22] is an extension to [23], and it replaced the decoder with an RNN. Reference [24] introduced a large Chinese dataset for short text summarization and it used a RNN based seq2seq model.”