Curriculum learning

By Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Aaron Edward Weston. 2009

7,325 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “C L ] 2 0 Ju n 20 vanced attention mechanisms [Xiong et al., 2017], attention networks [Vaswani et al., 2017], question answering [Seo et al., 2017, Xiong et al., 2018, Yu et al., 2016, Weissenborn et al., 2017], and curriculum learning [Bengio et al., 2009].”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “We highlight a few open problems along these lines: Task Scheduling: Scheduling tasks has been widely studied in the context of multitask learning and meta-learning, but remains relatively under-explored for multilingual NMT (Bengio et al., 2009; Pentina et al., 2015).”
Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Text-based heuristics such as word identity (Bengio et al., 2009) or input length (Spitkovsky et al., 2010; Gururangan et al., 2018) have sometimes been used as proxies for instance difficulty, but offer limited insight into difficulty w.r.t. models.”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, neutral): “Note that our training algorithm shares spirits with Curriculum Learning (Bengio et al., 2009) and SelfPaced Learning (Kumar et al.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “ed with neural rankers [Dehghani et al., 2017, MacAvaney et al., 2019b]. Also hoping to improve training, MacAvaney et al. [2020d] investigated whether monoBERT can beneﬁt from a training curriculum [Bengio et al., 2009] in which the model is presented with progressively more difﬁcult training examp”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “Notable variants of transfer learning are multitask learning [10], domain adaptation [6], and curriculum learning [8].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “more complex ones, which leads to a better performance in the final task. It has been shown in many previous works that appropriate curriculum strategies guide the learner towards better local minima [1]. Our idea is to gradually inject difficult information to the learner such that in the early sta”
Webly Supervised Learning of Convolutional Networks (2015, methodology): “After seventh layer, another fully connected layer is used to predict class labels.”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “learning [7] to guide the optimization process, where we begin optimizing the model to predict only two objects and increase this value by one once the validation loss plateaus.”
Emergence of Locomotion Behaviours in Rich Environments (2017, background): “The idea of curricula is long-standing in the machine learning literature (e.g. [30]).”
FaceNet: A unified embedding for face recognition and clustering (2015, background): “Choosing which triplets to use turns out to be very important for achieving good performance and, inspired by curriculum learning [1], we present a novel online negative exemplar mining strategy which ensures consistently increasing difficulty of triplets as the network trains.”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, background): “In addition, motivated by the fact that the larger the margin between the quality two selected version of the model, the easier for the comparative evaluator to learn to distinguish the training examples, we propose to use curriculum learning (Bengio et al. 2009) by feeding the comparative evaluator”
Learning to Execute (2014, methodology): “This strategy is has been examined in previous work on curriculum learning (Bengio et al., 2009).”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, methodology): “For these cases, we introduced a curriculum training [17] based approach, which incrementally adds each language after the model has been trained for a fixed number of iterations or the Character Error Rate (CER) goes below 50% for the previously added language.”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “In order to make training more stable, we experiment with a training curriculum (Bengio et al., 2009; Zaremba & Sutskever, 2014), which accelerates training and results in better generalization as well.”
Natural Language Processing (Almost) from Scratch (2011, methodology): “(Bengio et al., 2009) provides a more detailed discussion of this, the (as yet, poorly understood) “curriculum” process.”
Light Gated Recurrent Units for Speech Recognition (2018, background): “Moreover, the sorting approach exploits a curriculum learning strategy [62] that has been shown to slightly improve”
Representation Learning: A Review and New Perspectives (2012, background): “This is nicely related to the curriculum learning idea [24] that it may be much easier to learn simpler concepts first and then build”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “We found that this curriculum learning [6] approach is necessary to stabilize the training.”