Scaling Neural Machine Translation

By Myle Ott, Sergey Edunov, David Grangier, Michael Auli. 2018

647 citations recorded by the source corpus.

Sequence to sequence learning models still require several days to reach state of the art performance on large benchmark datasets using a single machine. This paper shows that reduced precision and large batch training can speedup training by nearly 5x on a single 8-GPU machine with careful tuning and implementation. On WMT’14 English-German translation, we match the accuracy of Vaswani et al. (2017) in under 5 hours when training on 8 GPUs and we obtain a new state of the art of 29.3 BLEU after training for 85 minutes on 128 GPUs. We further improve these results to 29.8 BLEU by training on the much larger Paracrawl dataset. On the WMT’14 English-French task, we obtain a state-of-the-art BLEU of 43.2 in 8.5 hours on 128 GPUs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “Past work in Neural Machine Translation has shown that training with very large mini-batches can both improve optimization speed and end-task performance when the learning rate is increased appropriately (Ott et al., 2018).”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “For multi-GPU setups, the cost of accumulating gradients is lower as it naturally helps to balance out uneven runtimes across workers (Ott et al., 2018).”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Gradient accumulation (Ott et al., 2018) is also applied to reduce multi-GPU communication overheads.”
Massively Multilingual Neural Machine Translation (2019, methodology): “We restrict our experiments to Transformer models (Vaswani et al., 2017) as they were shown to be very effective in recent benchmarks (Ott et al., 2018), also in the context of multilingual models (Lakew et al., 2018; Sachan and Neubig, 2018).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, neutral): “…utilization and beneﬁt faster computation, including meth-ods like gradient checkpointing and low precision training (Courbariaux et al., 2014; Ott et al., 2018), memory efﬁcient optimizers (Shazeer and Stern, 2018; Gupta et al., 2018) and frameworks supporting model parallelism (Shazeer et…”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “We train models with 16-bit ﬂoating point precision, following Ott et al. (2018).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “effective batch size but we found that models can still be trained effectively (Ott et al., 2018b).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “We found that large batch training is beneﬁcial for this dataset and we therefore accumulate gradient updates over two batches before committing a parameter update (Ott et al., 2018a).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “FP16 Training: Typical mixed-precision training recipes require storing model weights in both 16-bit (FP16) and 32-bit (FP32), as well as storing optimizer state in 32-bit to preserve accurate weight updates (Micikevicius et al., 2018; Ott et al., 2018).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, neutral): “We compare our results with two strong baselines reported by Ott et al. (2018); Li et al. (2019).”
Reformer: The Efficient Transformer (2020, methodology): “We additionally report detokenized BLEU scores as computed by sacreBLEU (Post, 2018). sacreBLEU Model BLEU Uncased3 Cased4 Vaswani et al. (2017), base model 27.”
Evaluating Gender Bias in Machine Translation (2019, methodology): “…both commercial and academic research: (1) Google Translate, 1 (2) Microsoft Translator, 2 (3) Amazon Translate, 3 (4) SYSTRAN, 4 (5) the model of Ott et al. (2018), which recently achieved the best performance on English-to-French translation on the WMT’14 test set, and (6) the model of Edunov…”
The State of Sparsity in Deep Neural Networks (2019, methodology): “This setup yielded a baseline BLEU score of 27.29 averaged across ﬁve runs.”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “In addition, significant research has gone into algorithmic approaches to efficiently scale training (Goyal et al., 2017; Ott et al., 2018) by reducing communication cost via compression (Alistarh et al.”
ELI5: Long Form Question Answering (2019, methodology): “Given our data length, we train with a large batch size by delaying gradient updates until a sufficient number of examples have been seen (Ott et al., 2018).”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “Empirically we observe better performance with a larger batch size as discussed in previous work (Liu et al., 2019; Ott et al., 2018), and thus we set the mini-batch size to 120.”
Learning Deep Transformer Models for Machine Translation (2019, background): “We solve this issue by accumulating gradients from two small batches (e.g. batch = 2048) before each update (Ott et al., 2018).”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “6 days [38]; the total batch size is 1.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “We test two tasks (1) language modeling on part of the RoBERTa corpus including Books (Zhu et al., 2015), CC-News (Nagel, 2016), OpenWebText (Gokaslan and Cohen, 2019), and CC-Stories (Trinh and Le, 2018); and (2) neural machine translation (NMT) (Ott et al., 2018) on WMT14+WMT16 (Macháˇcek and Boja”