Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

By Sébastien Forestier, Yoan Mollard, Pierre-Yves Oudeyer. 2017

209 citations recorded by the source corpus.

Intrinsically motivated spontaneous exploration is a key enabler of autonomous lifelong learning in human children. It allows them to discover and acquire large repertoires of skills through self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present the unsupervised multi-goal reinforcement learning formal framework as well as an algorithmic approach called intrinsically motivated goal exploration processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP algorithmic architecture relies on several principles: 1) self-generation of goals as parameterized reinforcement learning problems; 2) selection of goals based on intrinsic rewards; 3) exploration with parameterized time-bounded policies and fast incremental goal-parameterized policy search; 4) systematic reuse of information acquired when targeting a goal for improving other goals. We present a particularly efficient form of IMGEP that uses a modular representation of goal spaces as well as intrinsic rewards based on learning progress. We show how IMGEPs automatically generate a learning curriculum within an experimental setup where a real humanoid robot can explore multiple spaces of goals with several hundred continuous dimensions. While no particular target goal is provided to the system beforehand, this curriculum allows the discovery of skills of increasing complexity, that act as stepping stone for learning more complex skills (like nested tool use). We show that learning several spaces of diverse problems can be more efficient for learning complex skills than only trying to directly learn these complex skills. We illustrate the computational efficiency of IMGEPs as these robotic experiments use a simple memory-based low-level policy representations and search algorithm, enabling the whole system to learn online and incrementally on a Raspberry Pi 3.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “RR trains a set of policies, which is aligned with the population-based training in MARL (Jaderberg et al., 2017; 2019; Vinyals et al., 2019; Long et al., 2020; Forestier et al., 2017).”
Continual Learning for Robotics (2019, background): “This is generally done by a reward function that deﬁnes credit assignment [106], or it can also be deﬁned internally by more abstract rules such as self-supervision [54, 152], intrinsic motivation or curiosity [113, 142] as in [46, 27, 30, 78].”
Improving Intrinsic Exploration with Language Abstractions (2022, background): “Another long tradition generates curricula of intrinsic goals to encourage learning [7, 13, 14, 18–20 , Language for Exploration.”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[44] Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “There has also been effort putting to either explicitly or implicitly separate exploration and exploitation (Colas et al., 2018; Forestier et al., 2017; Levine et al., 2016).”
Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions (2020, methodology): “, 2017), intrinsically motivated goal exploration processes (IMGEPs) (Forestier et al., 2017), teacher-student curriculum learning (Matiisen et al.”
Grounding Language to Autonomously-Acquired Skills via Goal Generation (2021, unknown): “To facilitate robust estimation, LP is usually estimated on sets of goals with similar difﬁculty or similar dynamics (Forestier et al., 2017; Colas et al., 2019a).”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, unknown): “Again, only few of them have been applied to a real robot [126], and they have not been connected to representational redescription concerns yet, even if some preliminary work comes into this direction [74, 127].”
Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (2019, background): “Other ideas following contingency awareness include controllable feature learning [11, 20], tool use discovery [10] and sensing guidance [17, 18, 19].”