Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning

By Sanmit Narvekar, Jivko Sinapov, Peter Stone. 2017

123 citations recorded by the source corpus.

Transfer learning is a method where an agent reuses knowledge learned in a source task to improve learning on a target task. Recent work has shown that transfer learning can be extended to the idea of curriculum learning, where the agent incrementally accumulates knowledge over a sequence of tasks (i.e. a curriculum). In most existing work, such curricula have been constructed manually. Furthermore, they are fixed ahead of time, and do not adapt to the progress or abilities of the agent. In this paper, we formulate the design of a curriculum as a Markov Decision Process, which directly models the accumulation of knowledge as an agent interacts with tasks, and propose a method that approximates an execution of an optimal policy in this MDP to produce an agent-specific curriculum. We use our approach to automatically sequence tasks for 3 agents with varying sensing and action capabilities in an experimental domain, and show that our method produces curricula customized for each agent that improve performance relative to learning from scratch or using a different agent’s curriculum.

Subfields: Artificial Intelligence, Computer Science Applications, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Control and Systems Engineering

Citations in the corpus

From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “Narvekar et al. (2017a) proposed formulating the selection of tasks using a Curriculum Markov Decision Process (CMDP).”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “, 2017) or automatically generated curricula (Narvekar et al., 2017) – and hierarchical learning of real robot tasks (Forestier et al.”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, methodology): “Narvekar et al. (2017) denote the meta-MDP as a curriculum MDP (CMDP), where the state space S is the set of policies the learning agent can represent.”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “…from transfer learning (Taylor and Stone, 2009) could be extrapolated to multiagent scenarios such as curriculum learning (Svetlik et al., 2016; Narvekar et al., 2017) where existing techniques work for a single agents (independently) and therefore an open question is to reuse information from…”
Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification (2020, background): “The basic idea of curriculum learning is to organize samples or tasks in ascending order of difficulty, and it has been widely adopted for weakly supervised learning [25, 23, 13] reinforcement learning[41, 31, 32].”
A Survey on Curriculum Learning (2021, background): “…(CV) [31], [40], natural language processing (NLP) [86], [112], healthcare prediction [14], etc., various reinforcement learning (RL) tasks [20], [77], [93] as well as other applications such as graph learning [25], [88] and Curriculum learning is the training strategy that trains a machine…”
Learning Curriculum Policies for Reinforcement Learning (2018, methodology): “The exploration rate ϵ was set to 0.1, eligibility trace parameter λ to 0.9, and learning rate α to 0.1 (these values match those reported in [16]).”