Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation

By Dana Ruiter, Josef van Genabith, Cristina España-Bonet. 2020

14 citations recorded by the source corpus.

Self-supervised neural machine translation (SSNMT) jointly learns to identify and select suitable training data from comparable (rather than parallel) corpora and to translate, in a way that the two tasks support each other in a virtuous circle. In this study, we provide an in-depth analysis of the sampling choices the SSNMT model makes during training. We show how, without it having been told to do so, the model self-selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) performing a denoising curriculum. We observe that the dynamics of the mutual-supervision signals of both system internal representation types are vital for the extraction and translation performance. We show that in terms of the Gunning-Fog Readability index, SSNMT starts extracting and learning from Wikipedia data suitable for high school students and quickly moves towards content suitable for first year undergraduate students.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Radu Tudor Ionescu
2. Nicu Sebe
3. Paolo Rota
4. Petru Soviany
5. Fahad Shahbaz Khan
6. Christian Beecks

Citations in the corpus

• Neural Machine Translation for Low-resource Languages: A Survey (2021, unknown): “As a result, unsupervised [108], as well as self-supervised [159] NMT architectures, have been used to generate multilingual embeddings.”
• Curriculum Learning: A Survey (2021, background): “[142] analyze the behavior of self-supervised neural machine translation systems that jointly learn to select the right training data and to perform translation.”