Active learning of inverse models with intrinsically motivated goal exploration in robots

By Adrien Baranes, Pierre-Yves Oudeyer. 2013

458 citations recorded by the source corpus.

We introduce the Self-Adaptive Goal Generation Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsically motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills/policies that solve a corresponding distribution of parameterized tasks/goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy parameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters. We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: (1) learning the inverse kinematics in a highly-redundant robotic arm, (2) learning omnidirectional locomotion with motor primitives in a quadruped robot, and (3) an arm learning to control a fishing rod with a flexible wire. We show that (1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; (2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; (3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot.

Subfields: Artificial Intelligence, Control and Systems Engineering, Meta & Continual Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Meta & Continual Learning

In Robotics & Control

Citations in the corpus

Learning Parameterized Skills (2012, background): “Finally, Baranes and Oudeyer [2] proposed an active learning framework for acquiring parameterized skills based on competence progress.”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “kills being acquired become progressively more complex. Algorithms of intrinsic motivation collect data and acquire skills incrementally through the online (self-)generation of a learning curriculum (Baranes and Oudeyer, 2013; Forestier and Oudeyer, 2016). This allows the efﬁcient stochastic selecti”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “In the skill learning literature, Baranes et al.[1] have proposed a goal-driven active learning approach for learning skills in continuous sensorimotor spaces.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Autonomous acquisition of skills is related to the learning with intrinsic motivation, which considers the unsupervised setting where an agent must learn without extrinsic reward [39, 9, 26, 4, 3].”
Diversity is All You Need: Learning Skills without a Reward Function (2018, neutral): “Baranes & Oudeyer (2013).”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, methodology): “d. In our experiments, our models will be parameterized in a similar fashion. The novelty in this work is in how Alice deﬁnes the goal for Bob. The closest work to ours is that of Baranes and Oudeyer [1], who also have one part of the model that proposes tasks, while another part learns to complete ”
Visual Reinforcement Learning with Imagined Goals (2018, background): “Goal setting for policy learning has previously been discussed [3] and recently Péré et al.”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, methodology): “Depending on the algorithmic instantiation of this architecture, different Meta-Policy mechanisms can be used [3, 31].”
Reverse Curriculum Generation for Reinforcement Learning (2017, neutral): “idea and only tackle preliminary tasks [18]. A similar line of work uses intrinsic motivation based on learning progress to obtain “developmental trajectories” that focus on increasingly harder tasks [19]. Nevertheless, their method requires iteratively partitioning the full task space, which strong”
Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress (2012, background): “For instance, this has been demonstrated for learning in robot control [1] and options learning in hierarchical reinforcement learning [16] and has also been considered lately in machine learning under the name of curriculum learning [3].”
Reset-free Trial-and-Error Learning for Robot Damage Recovery (2016, background): “Active learning can alleviate this issue by exploring behaviors that improve the model “at the right place” [24].”
Learning by Asking Questions (2017, methodology): “Inspired by curriculum learning [5, 6, 29, 45], we use this difference to select questions on which the answering module can improve the fastest.”
Curiosity-Driven Development of Tool Use Precursors: a Computational Model (2016, background): “Keywords: curiosity-driven learning; tool use; goal babbling; overlapping waves; developmental trajectory; HACOB model”
CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning (2018, methodology): “This challenge can be approached within the framework of Intrinsically Motivated Goal Exploration Processes [1; 2], leveraging computational models of autonomous development in human infants [3].”