An Empirical Exploration of Curriculum Learning for Neural Machine Translation

By Xuan Zhang, Manish Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J. Martindale, Paul McNamee, Kevin Duh, Marine Carpuat. 2018

125 citations recorded by the source corpus.

Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presented during training to help train better models faster. We adopt a probabilistic view of curriculum learning, which lets us flexibly evaluate the impact of curricula design, and perform an extensive exploration on a German-English translation task. Results show that it is possible to improve convergence time at no loss in translation quality. However, results are highly sensitive to the choice of sample difficulty criteria, curriculum schedule and other hyperparameters.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “Curriculum methods based on difficulty and competence were explored in Zhang et al. (2018) and Platanios et al. (2019).”
Token-level Adaptive Training for Neural Machine Translation (2020, unknown): “In NMT, word frequency information is used for curriculum learning (Kocmi and Bojar, 2017; Zhang et al., 2018; Platanios et al., 2019) and domain adaptation data selection (Wang et al., 2017; Zhang and Xiong, 2018; Gu et al., 2019).”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “Other difﬁculty-orderings have been explored extensively in Zhang et al. (2018), showing that they, too, can speed-up training without a loss in translation performance.”
Curriculum Learning: A Survey (2021, methodology): “[26] perform an extensive analysis of curriculum learning on a German-English translation task.”
A Survey on Curriculum Learning (2021, methodology): “(vi) The performance of various predeﬁned Training Schedulers is sensitive to the initial learning rate (in NMT task) [138].”
Competence-based Curriculum Learning for Neural Machine Translation (2019, background): “Previous research has proposed various pooling operations, such as minimum, maximum, and average (Zhang et al., 2018), but they show that they do not work well in practice.”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “In the field of Natural Language Processing (NLP), criteria such as sentence length and term frequency are commonly utilized (Cirik et al., 2016; Zhang et al., 2018; Liu et al., 2018).”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, methodology): “Although there is no single criterion to measure difficulty for general neural machine translation tasks (Kocmi and Bojar, 2017; Wang et al., 2018; Zhang et al., 2018; Kumar et al., 2019; Platanios et al., 2019), for the domain adaptation scenario, we measure difficulty based on the distance from in”
Human-in-the-loop machine learning: a state of the art (2022, background): “Zhang et al. (2018) concluded that predefined CL is highly sensitive to the choices of difficult measurer and the hyperparameters.”
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (2019, background): “Two recent papers (Kocmi and Bojar, 2017; Zhang et al., 2018) explore similar ideas for NMT, and verify that this strategy can reduce training time and improve quality.”