Teacher–Student Curriculum Learning

By Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman. 2017

471 citations recorded by the source corpus.

We propose Teacher–Student Curriculum Learning (TSCL), a framework for automatic curriculum learning, where the Student tries to learn a complex task, and the Teacher automatically chooses subtasks from a given set for the Student to train on. We describe a family of Teacher algorithms that rely on the intuition that the Student should practice more those tasks on which it makes the fastest progress, i.e., where the slope of the learning curve is highest. In addition, the Teacher algorithms address the problem of forgetting by also choosing tasks where the Student’s performance is getting worse. We demonstrate that TSCL matches or surpasses the results of carefully hand-crafted curricula in two tasks: addition of decimal numbers with long short-term memory (LSTM) and navigation in Minecraft. Our automatically ordered curriculum of submazes enabled to solve a Minecraft maze that could not be solved at all when training directly on that maze, and the learning was an order of magnitude faster than a uniform sampling of those submazes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, background): “• Using a curriculum that gradually increases difficulty as training progresses simplifies training, since the problem is first solved on a single environment and additional environments are only added when a minimum level of performance is achieved [35, 67].”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, methodology): “Prior methods have studied curriculum generation by maintaining a separate goal setting policy or network (Sukhbaatar et al., 2017; Matiisen et al., 2017; Held et al., 2017).”
Learning quadrupedal locomotion over challenging terrain (2020, neutral): “Instead of directly using the reward function to evaluate the learning progress [25, 41–43], we evaluate c T s by the traversability of generated terrains, which is deﬁned as the success rate of traversing a terrain.”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, background): “In Teacher-Student Curriculum Learning [37], the teacher automatically selects sub-tasks that offer the highest slope of learning curve for the student.”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “In the context of reinforcement learning, this has been used to select tasks or task parameters (Kanitscheider et al., 2021; Matiisen et al., 2020; Portelas et al., 2020a).”
Transfer of Deep Reactive Policies for MDP Planning (2018, unknown): “, 2015], or by using a teacher network to propose a curriculum over tasks for effective multi-task learning [Matiisen et al., 2017].”
Adaptive Curriculum Learning (2021, background): “Moreover, curriculum learning is also prevalent in speech recognition [3, 21, 31], natural language [29, 33, 47], and reinforcement learning [10, 13, 22, 25, 26, 39, 40].”
Adaptive Scheduling for Multi-Task Learning (2019, background): “[14], where tasks with faster progress are preferred.”
NetTailor: Tuning the Architecture, Not Just the Weights (2019, background): “Other approaches use a teacher network to enhance the learning of a student network [12, 40].”
Automated curriculum generation for Policy Gradients from Demonstrations (2019, methodology): “We evaluated other methods for this, methods like Matiisen et al. [2017] (refer to 6 in Appendix), but determined that a simple method like this was enough.”
Automatic Curriculum Learning For Deep RL: A Short Survey (2020, unknown): “Both Matiisen et al. [2017] and Mysore et al. [2018] measure LP as the estimated derivative of the performance for each task in a discrete set (Minecraft mazes and Sonic the Hedgehog levels respectively) and apply a MAB algorithm to automatically build a curriculum for their learning agents.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “The curriculum can be explicitly generated by: searching the space (Schmidhuber, 2013), teacher-student setting (Matiisen et al., 2019), increasing distance between the starting point and goal (Jabri et al., 2019) or using a density model to generate a task distribution for the meta learner…”
Rapid locomotion via reinforcement learning (2022, methodology): “Curriculum learning has been leveraged for training robotic systems in the past [25, 28, 35, 41].”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Our work also relates to curriculum learning (Matiisen et al., 2020; Porte-las et al., 2019; Sukhbaatar et al., 2018; OpenAI et al., 2021; Campero et al., 2021; Fang et al., 2021; Mu et al., 2022), with the key difference that these methods typically have a speciﬁc downstream goal or task in mind.”
Curriculum Learning: A Survey (2021, methodology): “[65] introduce the Teacher-Student Curriculum Learning (TSCL) framework for reinforcement learning.”