Effective Approaches to Attention-based Neural Machine Translation

By Thang Luong, Hieu Pham, Christopher D. Manning. 2015

8,424 citations recorded by the source corpus.

An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Graham Neubig
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Yoshua Bengio
  7. Quoc V. Le
  8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Yi Yang

In Natural Language Processing

  1. Graham Neubig
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Luke S. Zettlemoyer
  5. Kyunghyun Cho
  6. Yejin Choi
  7. Percy Liang
  8. Richard Socher

In Model Architectures

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Kyunghyun Cho
  4. Yoav Goldberg
  5. Richard Socher
  6. Jianfeng Gao
  7. Mohit Bansal
  8. Graham Neubig

Citations in the corpus