Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
By Yonghui Wu, Mike Schuster, Z. Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey and 21 more. 2016
7,325 citations recorded by the source corpus.
Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Graham Neubig
- Kyunghyun Cho
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Jianfeng Gao
- Kyunghyun Cho
- Yejin Choi
- Percy Liang
- Noah A. Smith
In Machine Translation
- Yonatan Belinkov
- Felix Stahlberg
- Philipp Koehn
- Changhan Wang
- Ondrej Bojar
- Raj Dabre
- Ye Jia
- Nadir Durrani
Citations in the corpus
- Attention is All you Need (2017, methodology): “For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary [31].”
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “The specifics are: • We use WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary.”
- A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “This phenomenon has been observed with similar sequence generation tasks like image captioning with CIDEr (Ren-nie et al., 2016) and machine translation with BLEU (Wu et al., 2016; Norouzi et al., 2016).”
- Convolutional Sequence to Sequence Learning (2017, methodology): “…et al., 2016) propose a convolutional model based on characters without attention, with 30 layers in the encoder and 30 layers in the decoder, GNMT (Wu et al., 2016) represents the state of the art on this dataset and they use eight encoder LSTMs as well as eight decoder LSTMs, we quote their…”
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “While some work (Luong et al., 2015b; Zhou et al., 2016; Luong and Manning, 2016; Wu et al., 2016) has achieved state of the art results using deep networks, others (Jean et al., 2015; Chung et al., 2016; Sennrich et al., 2016b) have achieved similar results with far shallower ones.”
- Factorization tricks for LSTM networks (2017, methodology): “LSTM networks (Hochreiter & Schmidhuber, 1997) have been successfully used in language modeling (Jozefowicz et al., 2016; Shazeer et al., 2017), speech recognition (Xiong et al., 2016), machine translation (Wu et al., 2016), and many other tasks.”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “(Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et…”
- Neural Machine Translation in Linear Time (2016, methodology): “In our comparison we consider the following neural translation models: the Recurrent Continuous Translation Model (RCTM) 1 and 2 (Kalchbrenner & Blunsom, 2013); the RNN Enc-Dec (Sutskever et al., 2014; Cho et al., 2014); the RNN Enc-Dec Att with the attentional pooling mechanism (Bahdanau et al., 20”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “In this work, we consider attention-based neural machine translation (NMT) models Bahdanau et al. (2015); Luong et al. (2015), which have demonstrated excellent translation quality Wu et al. (2016), as the core models of our data augmentation pipeline.”
- MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “This is commonly done with BLEU but as shown by Wu et al. (2016), BLEU is not necessarily correlated with sample quality.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “6 [Wu et al., 2016] for the WMT translation and CNN/DM summarization tasks.”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “For Text Embedder , we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al., 2016).”
- Fine-Tuning Language Models from Human Preferences (2019, methodology): “Much of this work uses algorithmi-cally defined reward functions such as BLEU for translation (Ranzato et al., 2015; Wu et al., 2016), ROUGE for summarization (Ranzato et al., 2015; Paulus et al., 2017; Wu and Hu, 2018; Gao et al., 2019b), music theory-based rewards (Jaques et al., 2017), or event…”
- Massively Multilingual Neural Machine Translation (2019, background): “Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014; Sutskever et al., 2014) is the current state-ofthe-art approach for machine translation in both academia (Bojar et al., 2016, 2017, 2018) and industry (Wu et al., 2016; Hassan et al., 2018).”
- The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Until recently, the standard approach trained recurrent models with attention [Luong et al., 2015b, Bahdanau et al., 2014] on a single source-target language pair [Wu et al., 2016, Sennrich et al., 2017].”
- A Simple Method for Commonsense Reasoning (2018, background): “Although deep neural networks have achieved remarkable successes (e.g., [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), their dependence on supervised learning has been challenged as a significant weakness.”
- Generating Wikipedia by Summarizing Long Sequences (2018, methodology): “We then encode the text using sub-word tokenization similar to Wu et al. (2016) with a vocabulary size of 32,000 yielding tokenized input, x i : For various values of L in experiments, we truncate the tokens to form the input sequence: For the output, we use the same vocabulary and tokenization for…”