The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

By M. Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George F. Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen and 2 more. 2018

479 citations recorded by the source corpus.

The past year has witnessed rapid advances in sequence-to-sequence (seq2seq) modeling for Machine Translation (MT). The classic RNN-based approaches to MT were first out-performed by the convolutional seq2seq model, which was then out-performed by the more recent Transformer model. Each of these new approaches consists of a fundamental architecture accompanied by a set of modeling and training techniques that are in principle applicable to other seq2seq architectures. In this paper, we tease apart the new architectures and their accompanying techniques in two ways. First, we identify several key modeling and training techniques, and apply them to the RNN architecture, yielding a new RNMT+ model that outperforms all of the three fundamental architectures on the benchmark WMT’14 English to French and English to German tasks. Second, we analyze the properties of each fundamental seq2seq architecture and devise new hybrid architectures intended to combine their strengths. Our hybrid models obtain further improvements, outperforming the RNMT+ model on both benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, neutral): “Similar results were reported in Chen et al. (2018), where they found their hybrid model composed of a Transformer encoder and RNN decoder worked the best in machine translation.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017) have shown that recurrence is not essential for the task, but recurrence can contribute to the strongest models (Chen et al., 2018).”
Simple, Scalable Adaptation for Neural Machine Translation (2019, methodology): “Our model is identical to Vaswani et al. (2017), having 6 encoder and decoder layers (12 in total), except that we use hidden layers of size 8192 instead of 4096, and a learning rate schedule of (3.0, 40K) 4 , following Chen et al. (2018).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “The most effective approach has been to simply synthesize parallel data (Firat et al., 2016b; Chen et al., 2017, 2018b) and incorporate that into the training process.”
Fine-tune BERT for Extractive Summarization (2019, neutral): “The Transformer baseline has 6 layers, the hidden size is 512 and the feed-forward ﬁlter size is 2048 .”
Memory-Efficient Adaptive Optimization for Large-Scale Learning (2019, methodology): “We instead followed the experimental protocol described in a later work [8].”
Rethinking Attention with Performers (2020, background): “…of machine learning including natural language processing (NLP) (e.g. speech recognition (Luo et al., 2020)), neural machine translation (NMT) (Chen et al., 2018), document generation/summarization, time series prediction, generative modeling (e.g. image generation (Parmar et al., 2018)),…”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, methodology): “We use a larger version of Transformer Big containing 375M parameters (6 layers, 16 heads, 8192 hidden dimension) (Chen et al., 2018), and a shared source-target sentence-piece model (SPM) 1 (Kudo and Richardson, 2018) vocabulary with 64k individual tokens.”
Root Mean Square Layer Normalization (2019, methodology): “ious tasks ranging from computer vision [19, 26], speech recognition [37] to natural language processing [31, 35]. In some cases, LayerNorm was found to be essential for successfully training a model [6]. Besides, the decoupling from batch-based samples endows LayerNorm with the superiority over bat”
Learning Deep Transformer Models for Machine Translation (2019, background): “In principle, residual networks can be seen as instances of the ordinary differential equation (ODE), behaving like the forward Euler discretization with an initial value (Chang et al., 2018; Chen et al., 2018b).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “The set of patch tokens and [CLS] token are fed to a standard Transformer network with a “pre-norm” layer normalization [10, 32].”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “The encoder-decoder framework with attention has initially been introduced for machine translation where it dominates the ﬁeld now [27–29].”
Scaling Laws for Neural Machine Translation (2021, methodology): “We use a Hybrid model (with 380M parameters) (Chen et al., 2018) to back-translate (BT) these documents to English.”
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
Linguistic Knowledge and Transferability of Contextual Representations (2019, background): “Better understanding the differences between transformers and LSTMs is an active area of research (Chen et al., 2018; Tang et al., 2018), and we leave further exploration of these observations to future work.”
Fixup Initialization: Residual Learning Without Normalization (2019, background): “It was reported (Chen et al., 2018) that “Layer normalization is most critical to stabilize the training process.”