Competence-based Curriculum Learning for Neural Machine Translation

By Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnab'as P'oczos, Tom Michael Mitchell. 2019

425 citations recorded by the source corpus.

Current state-of-the-art NMT systems use large neural networks that are not only slow to train, but also often require many heuristics and optimization tricks, such as specialized learning rate schedules and large batch sizes. This is undesirable as it requires extensive hyperparameter tuning. In this paper, we propose a curriculum learning framework for NMT that reduces training time, reduces the need for specialized heuristics or large batch sizes, and results in overall better performance. Our framework consists of a principled way of deciding which training samples are shown to the model at different times during training, based on the estimated difficulty of a sample and the current competence of the model. Filtering training samples in this manner prevents the model from getting stuck in bad local optima, making it converge faster and reach a better solution than the common approach of uniformly sampling training examples. Furthermore, the proposed method can be easily applied to existing NMT models by simply modifying their input data pipelines. We show that our framework can help improve the training time and the performance of both recurrent neural network models and Transformers, achieving up to a 70% decrease in training time, while at the same time obtaining accuracy improvements of up to 2.2 BLEU.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “, 2019) learn a RL agent to schedule between different noise levels in the training data, while (Platanios et al., 2019) combined heuristics into a data curriculum.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “, 2019) (Platanios et al., 2019) (Ruiter et al.”
A Survey on Curriculum Learning (2021, background): “2, this function λ ( t ) must be monotone and non-decreasing, starting at λ (0) > 0 and ending at λ ( T ) = 1 This function is also called pacing function [33] or competence function [86] in literature.”
Curriculum Temperature for Knowledge Distillation (2022, background): “2018; Sinha, Garg, and Larochelle 2020) and natural language processing (Platanios et al. 2019; Tay et al. 2019).”
Adaptive Curriculum Learning (2021, background): “Moreover, curriculum learning is also prevalent in speech recognition [3, 21, 31], natural language [29, 33, 47], and reinforcement learning [10,13,22,25,26,39,40].”
Curriculum Learning: A Survey (2021, background): “[27] propose a continuous curriculum learning framework for neural machine translation.”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, background): “Although there is no single criterion to measure difficulty for general neural machine translation tasks (Kocmi and Bojar, 2017; Wang et al., 2018; Zhang et al., 2018; Kumar et al., 2019; Platanios et al., 2019), for the domain adaptation scenario, we measure difficulty based on the distance from in”
Human-in-the-loop machine learning: a state of the art (2022, background): “This mapping function can be a linear function (Hacohen and Weinshall 2019), a root function (Platanios et al. 2019), or even a geometric progression function (Penha and Hauff 2020).”
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (2019, background): “Work on meta-learning a curriculum originated with Tsvetkov et al. (2016), who used Bayesian optimization to learn a linear model for ranking examples in a word-embedding task.”
A Survey of Deep Active Learning (2020, background): “[164] proposes a curriculum learning framework related to AL for machine translation tasks.”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, methodology): “Platanios et al. (2019) link the introduction of difﬁcult samples with the NMT models’ competence .”
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering (2020, background): “The competence-aware curriculum in our work is related to curriculum learning [7,53,55,20,51,44,21,47] and machine teaching [61,62,38,11,40,15,58].”