Automated curricula through setter-solver interactions

By Sébastien Racanière, Andrew Kyle Lampinen, Adam Santoro, David P. Reichert, Vlad Firoiu, Timothy P. Lillicrap. 2019

60 citations recorded by the source corpus.

Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula--the breakdown of tasks into simpler, static challenges with dense rewards--to build up to complex behaviors. While curricula are also useful for artificial agents, hand-crafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Unlike prior methods that formulate an adversarial game around goal reaching [42, 36, 17], our method does not involve setting or reaching goals, but rather uses a single task to focus the forward policy, providing opportunities for the reset skills to learn varied adversarial perturbations.”
Situated Dialogue Learning through Procedural Environment Generation (2021, methodology): “Given this measure most methods either choose to adversarially set goals of steadily increasing difficulty (Sukhbaatar et al., 2018; Racaniere et al., 2019; Campero et al., 2021) or to maximize learning performance based on environment instances an agent finds difficult historically (Graves et al.,…”