Data and Parameter Scaling Laws for Neural Machine Translation

By Mitchell A. Gordon, Kevin Duh, Jared Kaplan. 2021

112 citations recorded by the source corpus.

We observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.

Subfields: Artificial Intelligence, Information Systems, Scaling & Learning Theory, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Scaling & Learning Theory

In Machine Translation

Citations in the corpus

Scaling Laws for Neural Machine Translation (2021, background): “[13] Mitchell A Gordon, Kevin Duh, and Jared Kaplan.”
Beyond neural scaling laws: beating power law scaling via data pruning (2022, background): “Empirically observed neural scaling laws [1, 2, 3, 4, 5, 6, 7, 8] in many domains of machine learning, including vision, language, and speech, demonstrate that test error often falls off as a power law with either the amount of training data, model size, or compute.”
Are Emergent Abilities of Large Language Models a Mirage? (2023, background): “Generating additional test data increases the resolution and reveals that even on Accuracy, the InstructGPT/GPT-3 family’s [4, 27] performance is above chance and improves in a smooth, continuous, predictable manner that qualitatively matches the mathematical model.”
KAN: Kolmogorov-Arnold Networks (2024, unknown): “NSLs are the phenomena where test losses behave as power laws against model size, data, compute etc [52, 53, 54, 55, 17, 56, 57, 58].”
SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, background): “A primary driver of recent success in machine learning has been the rise of self-supervised learning (SSL) scaled to ever larger models and unlabelled datasets [1, 2, 3, 4, 5, 6, 7, 8].”
Revisiting Neural Scaling Laws in Language and Vision (2022, methodology): “This has been used, for example, to estimate the required sample size in healthcare [9], neural machine translation (NMT) [19], and language models [23], among others [20, 22, 32].”