Training Tips for the Transformer Model

By Martin Popel, Ondrej Bojar. 2018

335 citations recorded by the source corpus.

Abstract This article describes our experiments in neural machine translation using the recent Tensor2Tensor framework and the Transformer sequence-to-sequence model (Vaswani et al., 2017). We examine some of the critical parameters that affect the final translation quality, memory usage, training stability and training time, concluding each experiment with a set of recommendations for fellow researchers. In addition to confirming the general mantra “more data and larger models”, we address scaling to multiple GPUs and provide practical tips for improved training regarding batch size, learning rate, warmup steps, maximum sentence length and checkpoint averaging. We hope that our observations will allow others to get better results given their particular hardware and data constraints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

One Epoch Is All You Need (2019, background): “One of the experiments of Popel & Bojar (2018) shows that larger neural machine translation (NMT) dataset results in higher BLEU with the same parameter budget and the same number of iterations, i.e., with smaller number of epochs.”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, background): “It has been shown that Transformer’s performance depends heavily on a batch size (Popel and Bojar, 2018), and we chose a large value of batch size to ensure that models show their best performance.”
On Layer Normalization in the Transformer Architecture (2020, methodology): “Popel & Bojar (2018) has shown that the ﬁnal model performance is quite sensitive to the value of the maximum learning rate and the number of warm-up iterations.”
On the Variance of the Adaptive Learning Rate and Beyond (2019, background): “However, it has been observed that these optimization methods may converge to bad/suspicious local optima, and have to resort to a warmup heuristic – using a small learning rate in the ﬁrst few epochs of training to mitigate such problem (Vaswani et al., 2017; Popel & Bojar, 2018).”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, methodology): “As Transformer was originally proposed as an NMT system [1], it has been widely studied on NMT tasks including hyperparameter search [9], parallelism implementation [5] and in comparison with RNN [10].”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, background): “Thus, when applying a replacement scheduler, a warm-up mechanism [25] is essentially adopted at the same time, which helps the training of a Transformer.”
CUNI Submission for Low-Resource Languages in WMT News 2019 (2019, background): “Popel and Bojar (2018) documented best practices to improve the performance of the model.”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, background): “It is acknowledged that Transformer model is extremely sensitive to the hyper-parameters such as the learning rate and the number of warmup steps (Popel and Bojar, 2018).”
The University of Cambridge’s Machine Translation Systems for WMT18 (2018, methodology): “Large batch training has received some attention in recent research (Smith et al., 2017; Neishi et al., 2017) and has been shown particularly useful for training the Transformer architecture with the Tensor2Tensor framework (Popel and Bojar, 2018).”
Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “Other methods of gradient clipping, such as choosing a step size proportional to the batch size, are not recommended for Transformers as these can lead to slower convergence [46].”
Competence-based Curriculum Learning for Neural Machine Translation (2019, methodology): “During training, we use a label smoothing factor of 0.1 (Wu et al., 2016) and the AMSGrad optimizer (Reddi et al., 2018) with its default parame- ters in TensorFlow, and a batch size of 5,120 tokens (due to GPU memory constraints).”
A Comprehensive Survey of Grammar Error Correction (2020, methodology): “It is worthy to mention that some training techniques including dropping out word embeddings of source words [95], checkpoint average [96], byte pair encoding (BPE) [97] and domain adaptation [53] are commonly used in training GEC systems.”
Understanding the Difficulty of Training Transformers (2020, background): “Moreover, comparing to other neural architectures, removing the warmup stage in Transformer training results in more severe consequences such as model divergence (Popel and Bojar, 2018; Liu et al., 2020a).”
Improving Transformer Optimization Through Better Initialization (2020, methodology): “Removing any of these components hampers optimization, and the model often fails to learn (Popel & Bojar, 2018).”
Neural Machine Translation: A Review (2019, methodology): “are naturally less noisy than gradients from small batches, and can yield better overall convergence [76, 126, 161].”
A Comprehensive Survey of Grammatical Error Correction (2021, methodology): “It is worthy to mention that some training tricks including dropping out word embeddings of source words [121], checkpoint average [103], byte pair encoding (BPE) [118], and domain adaptation [67] are commonly used in developing GEC systems.”