Reinforcement Learning based Curriculum Optimization for Neural Machine Translation

By Manish Kumar, George F. Foster, Colin Cherry, Maxim Krikun. 2019

79 citations recorded by the source corpus.

We consider the problem of making efficient use of heterogeneous training data in neural machine translation (NMT). Specifically, given a training dataset with a sentence-level feature such as noise, we seek an optimal curriculum, or order for presenting examples to the system during training. Our curriculum framework allows examples to appear an arbitrary number of times, and thus generalizes data weighting, filtering, and fine-tuning schemes. Rather than relying on prior knowledge to design a curriculum, we use reinforcement learning to learn one automatically, jointly with the NMT system, in the course of a single training run. We show that this approach can beat uniform baselines on Paracrawl and WMT English-to-French datasets by +3.4 and +1.3 BLEU respectively. Additionally, we match the performance of strong filtering baselines and hand-designed, state-of-the-art curricula.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “While most literature in multilingual NMT, including our work, relies on the same monolithic optimization approach for single and multitask models, 8 this choice might be far from optimal.”
Findings of the WMT 2019 Shared Task on Parallel Corpus Filtering for Low-Resource Conditions (2019, background): “58 Neural machine translation model training may combine data selection and model training, taking advantage of the increasing quality of the model to better detect noisy data or to increasingly focus on cleaner parts of the data (Wang et al., 2018; Kumar et al., 2019).”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, methodology): “To alleviate this cost, Kumar et al. (2019) use reinforcement learning on the pre-scored noisy corpus to jointly learn the denoising curriculum with NMT.”
Human-in-the-loop machine learning: a state of the art (2022, background): “Both traditional RL and Deep RL models are leveraged in these designs (Kumar et al. 2019; Matiisen et al. 2020), where the Deep RL models are stronger in performance but more time-consuming and harder to train.”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, background): “Although there is no single criterion to measure difficulty for general neural machine translation tasks (Kocmi and Bojar, 2017; Wang et al., 2018; Zhang et al., 2018; Kumar et al., 2019; Platanios et al., 2019), for the domain adaptation scenario, we measure difficulty based on the distance from in”
A Survey on Curriculum Learning (2021, methodology): “Beyond traditional RL algorithms, recent works also leverage deep RL models, e.g., Q-learning [53] and Deterministic Actor-Critic [140], to design RL Teacher methods for automatic data selection, sharing the same spirit with 7.”