CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

By Cédric Colas, Pierre-Yves Oudeyer, Olivier Sigaud, Pierre Fournier, Mohamed Chetouani. 2018

191 citations recorded by the source corpus.

In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to discover what is controllable in their environments, and what is not. Because some goals might prove easy and some impossible, agents must actively select which goal to practice at any moment, to maximize their overall mastery on the set of learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and 2) an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress. Agents focus sequentially on goals of increasing complexity, and focus back on goals that are being forgotten. Experiments conducted in a new modular-goal robotic environment show the resulting developmental self-organization of a learning curriculum, and demonstrate properties of robustness to distracting goals, forgetting and changes in body properties.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “This approach is extended in the CURIOUS algorithm [Colas et al., 2018].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, neutral): “, 2017) and modular (Colas et al., 2019) RL frameworks.”
Continual Learning for Robotics (2019, background): “As an example, manipulation tasks [72] such as grasping [121], reaching [123, 26], pushing buttons [84], throwing [155, 72] or stacking [26] objects (cubes, balls...) are common complex tasks built on comprehensive sets of experiments.”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “CURIOUS (Colas et al., 2019) sets a curriculum over environment goals by prioritising goal spaces which have shown recent learning progress and then sampling goals uniformly over goal spaces.”
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (2017, methodology): “Approaches such as Quality Diversity need 40M iterations for example for the learning of a hexapod’s locomotion (Cully et al., 2015), and deep reinforcement learning agents also require millions of training steps, e.g. 2M steps in the Atari game Montezuma’s Revenge (Kulkarni et al., 2016), or 50M in”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “91 Whereas handling abstract goals used to require engineers to hard-code speciﬁc goal representations and reward functions within the agent, 14,92,93 linguistic goals oﬀer abstraction via simple linguistic interactions.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “One approach in this setting explicitly modularizes the inputs to a neural network to handle each of the different goals, aided by multi-hot indicators of the active goals (Colas et al., 2019).”
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning (2021, background): “Multi-goal RL [47, 35, 13, 10] aims to deal with multiple tasks, learning to reach different goal states for each task.”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “One family of methods gives intrinsic reward for following a curriculum of goals [8, 12, 40].”
Efficient Exploration via State Marginal Matching (2019, methodology): “Most closely related to our work are methods that perform exploration in the space of states or goals (Colas et al., 2018; Held et al., 2017; Nair et al., 2018; Pong et al., 2019; Hazan et al., 2018).”