Scaling Laws for Neural Machine Translation

By Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier García, Ciprian Chelba, Colin Cherry. 2021

136 citations recorded by the source corpus.

We present an empirical study of scaling properties of encoder-decoder Transformer models used in neural machine translation (NMT). We show that cross-entropy loss as a function of model size follows a certain scaling law. Specifically (i) We propose a formula which describes the scaling behavior of cross-entropy loss as a bivariate function of encoder and decoder size, and show that it gives accurate predictions under a variety of scaling approaches and languages; we show that the total number of parameters alone is not sufficient for such purposes. (ii) We observe different power law exponents when scaling the decoder vs scaling the encoder, and provide recommendations for optimal allocation of encoder/decoder capacity based on this observation. (iii) We also report that the scaling behavior of the model is acutely influenced by composition bias of the train/test sets, which we define as any deviation from naturally generated text (either via machine generated or human translated text). We observe that natural text on the target side enjoys scaling, which manifests as successful reduction of the cross-entropy loss. (iv) Finally, we investigate the relationship between the cross-entropy loss and the quality of the generated translations. We find two different behaviors, depending on the nature of the test data. For test sets which were originally translated from target language to source language, both loss and BLEU score improve as model size increases. In contrast, for test sets originally translated from source language to target language, the loss improves, but the BLEU score stops improving after a certain threshold. We release generated text from all models used in this study.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “…of trained language models as they scale (Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Mikami et al., 2021; Pu et al., 2021; Sharma & Kaplan, 2020; Ghorbani et al., 2021), but prior work connecting these “Scaling Laws” to the learning dynamics of language models is minimal.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “Recent research has shown that training on datasets of mixed human and machine-generated data can significantly impair the performance of translation systems (Ghorbani et al., 2021).”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “We ran experiments on two common benchmarks: IWSLT 2014 (Cettolo et al., 2014), and WMT 2020 (Koehn et al., 2020), as well as an internal benchmark dataset which we call Web Domain (a version of this dataset was previously used by Ghorbani et al. (2021)).”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, background): “…in Neural Machine Translation, where cross entropy loss can decline while the quality of the translation (measured by BLEU score) remains constant [16], and in contrastive representation learning, where lower capacity models have worse losses on the test set but still learn better representations…”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “…investigate the scaling of neural networks to better understand scaling laws and optimize training budget (Hest-ness et al., 2017; Kaplan et al., 2020; Bornschein et al., 2020; Ghorbani et al., 2021; Bahri et al., 2021; Tay et al., 2021; Rae et al., 2021; Hoffmann et al., 2022; Ivgi et al., 2022).”
Revisiting Neural Scaling Laws in Language and Vision (2022, background): “While theoretical arguments alone seldom predict scaling law parameters in modern neural architectures [2, 21, 32], it has been observed that the benefit of scale could be predicted empirically [3, 4, 9, 17, 18, 20, 22, 23, 28, 30, 31].”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “Scaling up model capacity is an obvious yet effective approach for better performance in natu-ral language processing (NLP) tasks (Brown et al., 2020; Kaplan et al., 2020; Ghorbani et al., 2021; Zhou et al., 2020b).”
Scaling Data-Constrained Language Models (2023, background): “…minimize the loss ( L ) subject to a compute resource constraint ( C ) via optimal allocation to N and D as: Currently, there are established best practices for scaling LLMs. Return follows a power-law: loss scales as a power-law with the amount of compute used for training [39, 46, 6, 35, 7, 41].”