Massive Exploration of Neural Machine Translation Architectures

By Denny Britz, Anna Goldie, Minh-Thang Luong, Quoc V. Le. 2017

546 citations recorded by the source corpus.

Neural Machine Translation (NMT) has shown remarkable progress over the past few years, with production systems now being deployed to end-users. As the field is moving rapidly, it has become unclear which elements of NMT architectures have a significant impact on translation quality. In this work, we present a large-scale analysis of the sensitivity of NMT architectures to common hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on a WMT English to German translation task. Our experiments provide practical insights into the relative importance of factors such as embedding size, network depth, RNN cell type, residual connections, attention mechanism, and decoding heuristics. As part of this contribution, we also release an open-source NMT framework in TensorFlow to make it easy for others to reproduce our results and perform their own experiments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Learning Deep Transformer Models for Machine Translation (2019, methodology): “Deep models have been explored in the context of neural machine translation since the emergence of RNN-based models.”
Relation Networks for Object Detection (2017, methodology): “Our approach is motivated by the success of attention modules in natural language processing field [5, 49].”
NeMo: a toolkit for building AI applications using Neural Modules (2019, background): “These toolkits could be loosely classiﬁed into two main groups: (1) higher-level neural network APIs such as Keras[11], Sonnet [18], PyTorch Ignite [3], PyTorch Lightning [4] and (2) conﬁguration-driven tookits such as Tensor2Tensor [20], Ludwig [5], OpenSeq2Seq [13], FairSeq [16], OpenNMT [12], Seq”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “Two systems are based on the tfseq2seq toolkit (Britz et al. 2017): NLE (Agarwal, Dymetman, and Gaussier 2018) built a character-to-character seq2seq (using simply characters of the original MR as inputs), TR1 (Schilder et al.”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “Residual connections [16] and dense connections [19] were adopted from computer vision architectures and impacted on machine translation [7], text classification [43], and machine reading comprehension [40].”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “Source and target tokens are processed into 37K shared sub-word units based on byte-pair encoding (BPE) (Britz et al., 2017).”
Supervised Contrastive Learning (2020, background): “Deep network training is well known to be sensitive to hyper-parameters and a large body of literature is devoted to finding efficient ways to perform hyperparameter tuning [2, 8, 3].”
Deformable ConvNets V2: More Deformable, Better Results (2018, background): “Relation Networks and Attention Modules are first proposed in natural language processing [13, 14, 3, 35] and physical system modeling [2, 37, 22, 34, 9, 31].”
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension (2017, background): “A commonly used function is multiplicative attention (Britz et al., 2017) , where U, V ∈ R k × d h , and k is the attention hidden size.”
Training Deeper Neural Machine Translation Models with Transparent Attention (2018, background): “This increase in inter-connectivity significantly shortens the creditassignment path (Britz et al., 2017), making the network less susceptible to optimization pathologies like vanishing gradients.”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, methodology): “Following standard settings [Britz et al., 2017] we use a bidirectional LSTM as encoder and an 4-layer LSTM with attention [Bahdanau et al.”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, methodology): “Sentences are encoded using byte-pair encoding [4], with a shared source-target vocabulary of about 37k tokens.”
Optimal Completion Distillation for Sequence Learning (2018, background): “Some papers (e.g. Britz et al. (2017)) have investigated the use of LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Cho et al., 2014) cells, while others proposed new architectural components such as soft attention (Bahdanau et al., 2015), convolution (Gehring et al., 2017), and self-attention…”
Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization (2018, background): “Recent development of neural networks has witnessed the success of Neural Machine Translation (NMT), which has achieved state-of-the-art performance (Bahdanau, Cho, and Bengio 2015; Britz et al. 2017; Gehring et al. 2017) through end-to-end learning.”
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation (2018, methodology): “); } buggy code public boolean METHOD_1 () { return (time()) >= (VAR_1); } Þxed code <eos> <eos> <eos> Fig. 4. Beam Search Visualization. into account both past and future inputs [10]. That is, each state h i represents the concatenation (dashed box in Fig.1) of the states produ”
OpenSeq2Seq: Extensible Toolkit for Distributed and Mixed Precision Training of Sequence-to-Sequence Models (2018, background): “Among the most popular ones are: OpenNMT (Klein et al., 2017), Seq2Seq (Britz et al., 2017), NMT (Luong et al., 2017), and Tensor2Tensor (Vaswani et al., 2017).”
Stronger Baselines for Trustable Results in Neural Machine Translation (2017, background): “of generally being “good enough” have made it a popular choice for researchers and NMT toolkit authors6 (Arthur et al., 2016; Lee et al., 2016; Britz et al., 2017; Sennrich et al., 2017).”