Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

By Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, Wei Xu. 2016

224 citations recorded by the source corpus.

Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years. However, most of the existing NMT models are shallow and there is still a performance gap between a single NMT model and the best conventional MT system. In this work, we introduce a new type of linear connections, named fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks, and an interleaved bi-directional architecture for stacking the LSTM layers. Fast-forward connections play an essential role in propagating the gradients and building a deep topology of depth 16. On the WMT’14 English-to-French task, we achieve BLEU=37.7 with a single attention model, which outperforms the corresponding single shallow model by 6.2 BLEU points. This is the first time that a single NMT model achieves state-of-the-art performance and outperforms the best conventional model by 0.7 BLEU points. We can still achieve BLEU=36.3 even without using an attention mechanism. After special handling of unknown words and model ensembling, we obtain the best score reported to date on this task with BLEU=40.4. Our models are also validated on the more difficult WMT’14 English-to-German task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, neutral): “2 · 10(20) Deep-Att + PosUnk Ensemble [32] 40.”
Convolutional Sequence to Sequence Learning (2017, methodology): “Models with many layers often rely on shortcut or residual connections (He et al., 2015a; Zhou et al., 2016; Wu et al., 2016).”
Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “We expected deep models to perform better (Zhou et al., 2016; Szegedy et al., 2015) across the board, and we believe that our experiments demonstrate the need for more robust techniques for optimizing deep sequential models.”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “…6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2 (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08…”
Neural Machine Translation in Linear Time (2016, background): “In our comparison we consider the following neural translation models: the Recurrent Continuous Translation Model (RCTM) 1 and 2 (Kalchbrenner & Blunsom, 2013); the RNN Enc-Dec (Sutskever et al., 2014; Cho et al., 2014); the RNN Enc-Dec Att with the attentional pooling mechanism (Bahdanau et al., 20”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “The decoder RNN network produces a hidden state yi for the next symbol to be predicted, which then goes through the softmax layer to generate a probability distribution over candidate output symbols.”
Sequence-Level Knowledge Distillation (2016, background): “…NMT systems typically need to be very high capacity in order to perform well: Sutskever et al. (2014) used a 4 -layer LSTM with 1000 hidden units per layer (herein 4 × 1000 ) and Zhou et al. (2016) obtained state-of-the-art results on English → French with a 16 -layer LSTM with 512 units per layer.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…2014; Bahdanau et al., 2014) have been widely adopted as the state-of-the-art approach for machine translation, both in the research community (Bojar et al., 2016a, 2017, 2018b) and for large-scale production systems (Wu et al., 2016; Zhou et al., 2016; Crego et al., 2016; Hassan et al., 2018).”
Learning Deep Transformer Models for Machine Translation (2019, neutral): “To ease optimization, researchers tried to reduce the number of non-linear transitions (Zhou et al., 2016; Wang et al., 2017).”
HyperNetworks (2016, methodology): “Model Test BLEU Log Perplexity Deep-Att + PosUnk (Zhou et al., 2016) 39.”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “2014), often with residual or skip connections (Wu et al., 2016; Zhou et al., 2016) to enable stacking of several layers (§2).”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, methodology): “We evaluate our models using the standard BLEU score metric and to make our results comparable to [27, 19, 31, 29], we report tokenized BLEU score as computed by the multi-bleu.”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “Seq2seq systems have shown strong results in machine translation (Zhou et al., 2016).”
What do Neural Machine Translation Models Learn about Morphology? (2017, background): “Modern NMT systems use very deep architectures with up to 8 or 16 layers (Wu et al., 2016; Zhou et al., 2016).”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “The RNN-based NMT approach, or RNMT, was quickly established as the de-facto standard for NMT, and gained rapid adoption into large-scale systems in industry, e.g. Baidu (Zhou et al., 2016), Google (Wu et al., 2016), and Systran (Crego et al., 2016).”
Training Deeper Neural Machine Translation Models with Transparent Attention (2018, background): “The first major improvement in NMT quality since the switch to neural models, amongst other factors, was brought about by a huge scale up in model capacity (Zhou et al., 2016; Wu et al., 2016).”
SYSTRAN's Pure Neural Machine Translation Systems (2016, methodology): “Ensemble decoding has been verified as a practical technique to further improve the performance compared to a single Encoder-Decoder model (Sennrich et al., 2016b; Wu et al., 2016; Zhou et al., 2016).”