Automated Curriculum Learning for Neural Networks

By Alex Graves, Marc G. Bellemare, Jacob Menick, R´emi Munos, Koray Kavukcuoglu. 2017

642 citations recorded by the source corpus.

We introduce a method for automatically selecting the path, or syllabus, that a neural network follows through a curriculum so as to maximise learning efficiency. A measure of the amount that the network learns from each data sample is provided as a reward signal to a nonstationary multi-armed bandit algorithm, which then determines a stochastic syllabus. We consider a range of signals derived from two distinct indicators of learning progress: rate of increase in prediction accuracy, and rate of increase in network complexity. Experimental results for LSTM networks on three curricula demonstrate that our approach can significantly accelerate learning, in some cases halving the time required to attain a satisfactory performance level.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “On the other hand, dynamic or curriculum learning strategies reﬁne the ratio of tasks simultaneously with training, based on metrics derived from the current state of the learner (Graves et al., 2017).”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “s of curriculum learning is highly sensitive with respect to the modality of progression through the tasks. Furthermore, this approach assumes that tasks can be ordered by a single axis of difﬁculty. Graves et al. (2017) proposed to treat the task selection problem as a stochastic policy over the ta”
Solving Rubik's Cube with a Robot Hand (2019, background): “• Using a curriculum that gradually increases difficulty as training progresses simplifies training, since the problem is first solved on a single environment and additional environments are only added when a minimum level of performance is achieved [35, 67].”
Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Specifically, prediction gain (PG) (Bellemare et al., 2016; Graves et al., 2017) picks the curriculum according to L((x?, y?);θ)−L((x?, y?);θ), where θ′ is the parameter after the network updates θ for one step using (x?, y?).”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, background): “As argued in curriculum learning literature [6, 13], easy or clean examples are favored in the early stage of training of a model.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “Curriculum learning has been shown to speed up the training of neural networks, while also leading to better generalization (Bengio et al., 2009; Graves, Bellemare, Menick, Munos, & Kavukcuoglu, 2017; Hunziker et al., 2019; Zhou & Bilmes, 2018; Fan, Tian, Qin, Li, & Liu, 2018; Achille, Rovere, &…”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Compared with curriculum learning [12,13], the learner does not control the task ordering.”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “This motivates the automated curriculum learning setting (Bengio et al., 2009) as recently explored for multi-task learning in (Graves et al., 2017) or rather automated generative curriculum learning in our case.”
Hindsight Experience Replay (2017, background): “Graves et al. (2017) consider a setup where there is a ﬁxed discrete set of tasks and empirically evaluate different strategies for automatic curriculum generation in this settings.”
Meta-Transfer Learning for Few-Shot Learning (2018, background): “[3] and is popular for multi-task learning [35, 42, 55, 14].”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, background): “Multitask learning has also been explored in the context of curriculum learning (Graves et al., 2017), where subsets of tasks are subsequently trained based on local rewards; we here explore the opposite approach, where tasks are jointly trained based on global rewards such as total loss decrease.”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “Curriculum-based approaches with manually designed schedules have been explored in supervised learning [8, 9, 10, 11] to split particularly complex tasks into smaller, easier-to-solve sub-problems.”
Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, background): “When contexts/tasks are not sampled uniformly but shaped into curricula based on recent learning progress [8, 11], this ampliﬁes the winner-take-all dynamics.”
Meta-Transfer Learning Through Hard Tasks (2019, background): “[34] and is popular for multitask learning [59]–[61].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, background): “Hints can be provided to the agent which bias it towards promising regions of the state space either via reward-shaping (Ng et al., 1999) or by introducing a sequence of curriculum tasks (Bengio et al., 2009; Graves et al., 2017).”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “A number of methods attempt to use learning progress (Kaplan and Oudeyer, 2007; Schmidhuber, 2010) on a task as a way to decide whether the task should be trained on or not (Graves et al., 2017).”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, background): “, 2018), reinforcement learning (Graves et al., 2017; Glover and Hokamp, 2019; Xu et al., 2019), transfer learning (Zhang et al.”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, background): “(Supancic & Ramanan, 2013; Chen & Gupta, 2015), natural language processing (Turian et al., 2010), multitask learning (Graves et al., 2017).”
Situated Dialogue Learning through Procedural Environment Generation (2021, methodology): “, 2021) or to maximize learning performance based on environment instances an agent finds difficult historically (Graves et al., 2017; Portelas et al., 2020).”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, background): “In addition, it could be augmented by utilizing the rest of the AVA dataset movies to provide large amounts of unsupervised or semi-supervised data in conjunction with curriculum learning [4, 23] or reinforcement learning [30, 31].”