Source Task Creation for Curriculum Learning

By Sanmit Narvekar, Jivko Sinapov, Matteo Leonetti, Peter Stone. 2016

130 citations recorded by the source corpus.

Transfer learning in reinforcement learning has been an active area of research over the past decade. In transfer learning, training on a source task is leveraged to speed up or otherwise improve learning on a target task. This paper presents the more ambitious problem of curriculum learning in reinforcement learning, in which the goal is to design a sequence of source tasks for an agent to train on, such that final performance or learning speed is improved. We take the position that each stage of such a curriculum should be tailored to the current ability of the agent in order to promote learning new behaviors. Thus, as a first step towards creating a curriculum, the trainer must be able to create novel, agent-specific source tasks. We explore how such a space of useful tasks can be created using a parameterized model of the domain and observed trajectories on the target task. We experimentally show that these methods can be used to form components of a curriculum and that such a curriculum can be used successfully for transfer learning in 2 challenging multiagent reinforcement learning domains.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Selective Experience Replay for Lifelong Learning (2018, background): “The order in which tasks are encountered does impact learning, and several groups have investigated the effects of ordering (Bengio et al. 2009; Ruvolo and Eaton 2013a; Narvekar et al. 2016).”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “Narvekar et al. (2016) ﬁrstly applied CL to RL and proposed a new CL framework.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Through the steady increase within the curriculum, trained policies can perform better than without a curriculum (Gupta et al. 2017; Long et al. 2020; Narvekar et al. 2016).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Such methods often utilize curricula provided by human designers [28, 322, 300], though several recent methods seek to automatically generate a curriculum [273, 374, 115].”
Learning Curriculum Policies for Reinforcement Learning (2018, methodology): “This insight is the basis for curriculum learning [4, 15].”
An Introduction to Deep Reinforcement Learning (2018, unknown): “The idea is to start by learning small and easy aspects of the target task and then to gradually increase the difficulty level (Bengio et al., 2009; Narvekar et al., 2016).”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, methodology): “ transferability matrix for all pairs of tasks, and create a curriculum by chaining tasks backwards from the target tasks greedily with respect to it. 3) Extract a feature vector for each task (as in Narvekar et al. (2016)), 22 Curriculum Learning for Reinforcement Learning Domains and learn a regre”
Learning symmetric and low-energy locomotion (2018, background): “…in machine learning have shown that CL can improve performance and efficiency of learning problems such as question-answering [Graves et al. 2017], classification [Bengio et al. 2009; Pentina et al. 2015], navigation [Held et al. 2017; Matiisen et al. 2017] and game playing [Narvekar et al. 2016].”
Boosted Curriculum Reinforcement Learning (2022, methodology): “When not assuming any additional information of the aforementioned forms in a value-or Q -function transfer setting, methods that extend upon the naive idea of initializing the Q -function of the next task with the most recently learned one, as e.g. done in Narvekar et al. (2016), are rather sparse.”