Stronger Baselines for Trustable Results in Neural Machine Translation

By Michael J. Denkowski, Graham Neubig. 2017

119 citations recorded by the source corpus.

Interest in neural machine translation has grown rapidly as its effectiveness has been demonstrated across language and data scenarios. New research regularly introduces architectural and algorithmic improvements that lead to significant gains over “vanilla” NMT implementations. However, these new techniques are rarely evaluated in the context of previously published techniques, specifically those that are widely used in state-of-the-art production and shared-task systems. As a result, it is often difficult to determine whether improvements from research will carry over to systems deployed for real-world use. In this work, we recommend three specific methods that are relatively easy to implement and result in much stronger experimental systems. Beyond reporting significantly higher BLEU scores, we conduct an in-depth analysis of where improvements originate and what inherent weaknesses of basic NMT models are being addressed. We then compare the relative gains afforded by several other techniques proposed in the literature when starting with vanilla systems versus our stronger baselines, showing that experimental conclusions may change depending on the baseline chosen. This indicates that choosing a strong baseline is crucial for reporting reliable experimental results.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Natural Language Processing

Citations in the corpus

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “NMT has shown remarkable results in several shared tasks (Denkowski and Neubig, 2017; Nakazawa et al., 2017), and its effective approach has had a strong influence on other related NLP tasks such as dialog generation (Vinyals and Le, 2015) and automatic summarization (Rush et al.”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “, 2016) is a de facto standard subword segmentation algorithm applied to many NMT systems and achieving top translation quality in several shared tasks (Denkowski and Neubig, 2017; Nakazawa et al., 2017).”
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (2018, methodology): “In (Denkowski and Neubig, 2017) the authors recommend three specific techniques for strengthening NMT systems and empirically demonstrated how incorporating those techniques improves the reliability of the experimental results.”
When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (2018, neutral): “5 when development loss decreases (Denkowski and Neubig, 2017).”
Rapid Adaptation of Neural Machine Translation to New Languages (2018, methodology): “Following standard practice (Sennrich et al., 2016; Denkowski and Neubig, 2017), we break low-frequency words into subwords using the sentencepiece toolkit.”
Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration (2018, methodology): “Thus, “adaptive gradient” algorithms such as RMSProp (Algorithm 2) [39] and ADAM (Algorithm 3) [20] have become very popular for optimizing deep neural networks [28, 44, 10, 14, 34, 4, 21].”
Sockeye: A Toolkit for Neural Machine Translation (2017, unknown): “Learning schedules While optimizers such as Adam and Eve dynamically adjust the step size for each update, recent work has shown the value of annealing the base learning rate α throughout training [Vaswani et al., 2017, Denkowski and Neubig, 2017].”
SHAPE: Shifted Absolute Position Embedding for Transformers (2021, background): “Building a strong baseline is essential for trustable results (Denkowski and Neubig, 2017).”
A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation (2018, methodology): “The number of BPE segmentation rules is set to 8 , 000 , following the suggestion of Denkowski and Neubig (2017) for experiments in small training data condition.”
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (2018, background): “In this work, we make the argument that such algorithms make strong baselines for any task in line with recent efforts highlighting the usefulness of classic approaches (Melis et al., 2017; Denkowski and Neubig, 2017).”
Correcting Length Bias in Neural Machine Translation (2018, background): “Most of the experimental settings below follow the recommendations of Denkowski and Neubig (2017).”
Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary (2018, methodology): “Following [30], the number of BPE segmentation rules is set to 8 , 500 for the data size used in our experimental setting.”
Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, unknown): “Following the recommendation in [21], the segmentation rules are set to 8k.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “Another study, by Denkowski and Neubig [259], examined a number of other techniques, recommending three: using Adam optimization, restarting multiple times, with learning rate annealing; performing subword translation; and using an ensemble of decoders.”
An Empirical Study of Building a Strong Baseline for Constituency Parsing (2018, result): “Our motivation is basically identical to that described in Denkowski and Neubig (2017).”
Toward Robust Neural Machine Translation for Noisy Input Sequences (2017, background): “0003, restarting Adam and halving learning rates when perplexities did not improve for 2 consecutive epochs [36].”
Multi-layer Representation Fusion for Neural Machine Translation (2018, methodology): “After that, we restart Adam and continue the training for additional 20 epochs with a ﬁxed learning rate 5 e − 5 and a smaller mini-batch of 32 (Denkowski and Neubig, 2017).”