Automatic Curriculum Learning For Deep RL: A Short Survey

By Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer. 2020

235 citations recorded by the source corpus.

Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL). These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. To do so, ACL mechanisms can act on many aspects of learning problems. They can optimize domain randomization for Sim2Real transfer, organize task presentations in multi-task robotic settings, order sequences of opponents in multi-agent scenarios, etc. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.

Subfields: General Health Professions, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Philosophy

1. Jack Parker-Holder
2. Pierre-Yves Oudeyer
3. Edward Grefenstette
4. Yuke Zhu

Citations in the corpus

• Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “See (Portelas et al., 2020) for a survey of automated curriculum learning approaches for deep RL.”
• Replay-Guided Adversarial Environment Design (2021, background): “In inducing parallel curricula, DCD follows a rich lineage of curriculum learning methods [2, 32, 28, 23].”
• Grounding Language to Autonomously-Acquired Skills via Goal Generation (2021, unknown): “(Portelas et al., 2020) inspired from the CURIOUS algorithm (Colas et al.”
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (2017, background): “See Portelas et al. (2020) for a review of other recent curriculum learning methods applied to reinforcement learning scenarios.”
• Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[43] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer.”
• Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, background): “This is a common setting in the recent Automatic Curriculum Learning (ACL) literature [4, 25].”
• A Survey on Curriculum Learning (2021, methodology): “Automatic CL is also broadly applied to Deep RL tasks, and we refer readers to the recent surveys [76], [87] for further reading.”