Curiosity Driven Exploration of Learned Disentangled Goal Spaces

By Adrien Laversanne-Finot, Alexandre Péré, Pierre-Yves Oudeyer. 2018

90 citations recorded by the source corpus.

Intrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors, an efficient exploration requires that the structure of the goal space reflects the one of the environment. In this paper we show that using a disentangled goal space leads to better exploration performances than an entangled goal space. We further show that when the representation is disentangled, one can leverage it by sampling goals that maximize learning progress in a modular manner. Finally, we show that the measure of learning progress, used to drive curiosity-driven exploration, can be used simultaneously to discover abstract independently controllable features of the environment.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “sequence modeling, curiosity-based exploration or even domain adaptation in reinforcement learning [45, 25, 37, 16, 18, 29, 47].”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “More recently, some models (Haber et al., 2018; Laversanne-Finot et al., 2018) have shown interesting exploration behaviors emerge through task-free curiosity-driven exploration in agents.”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “While prior work (Steenbrugge et al., 2018; Laversanne-Finot et al., 2018; Nair et al., 2018; Higgins et al., 2017b; 2018) successfully applied disentanglement methods such as β-VAE on a variety of downstream tasks, it is not clear to us that these approaches and trained models performed well…”
Towards a Definition of Disentangled Representations (2018, background): “For example, subsequent tasks may safely ignore certain parts of such a representation to form abstractions [41, 53], or use novel recombinations of values within the subspaces for improved generalisation [40].”
Continual Learning for Robotics (2019, background): “This is generally done by a reward function that deﬁnes credit assignment [106], or it can also be deﬁned internally by more abstract rules such as self-supervision [54, 152], intrinsic motivation or curiosity [113, 142] as in [46, 27, 30, 78].”
Disentangling Factors of Variation Using Few Labels (2019, background): “These are not limiting assumption especially for sequential data or reinforcement learning [61, 59, 38, 46, 21, 22].”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, background): “The idea that a good representation can help with such shortcomings is not new, and recently a number of papers have demonstrated that models with disentangled representations show improvements in terms of these shortcomings (Higgins et al., 2017b; 2018b; Achille et al., 2018; Steenbrugge et al., 20”
Contrastive Learning of Structured World Models (2019, methodology): “…representation learning in environments similar to ours is often approached by models based on autoencoders (Corneil et al., 2018; Watter et al., 2015; Ha & Schmidhuber, 2018; Hafner et al., 2019; Laversanne-Finot et al., 2018) or via adversarial learning (Kurutach et al., 2018; Wang et al., 2019).”
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States (2021, background): “There has also been several empirical works that either focus on the factored state space setting (e.g., Kim & Mnih (2018); Thomas et al. (2018); Laversanne-Finot et al. (2018); Miladinović et al. (2019)), or the factored action space setting (e.g., He et al. (2016); Sharma et al. (2017)).”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, unknown): “Again, only few of them have been applied to a real robot [126], and they have not been connected to representational redescription concerns yet, even if some preliminary work comes into this direction [74, 127].”
CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning (2018, background): “This challenge can be approached within the framework of Intrinsically Motivated Goal Exploration Processes [1; 2], leveraging computational models of autonomous development in human infants [3].”
Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (2017, background): “The features may be given to the agent, or learned, for instance using generative models (see Péré et al. (2018); Nair et al. (2018); Laversanne-Finot et al. (2018); Reinke et al. (2020)).”
Learning Actionable Representations with Goal-Conditioned Policies (2018, background): “Several works have sought to learn representations that are speciﬁcally suited for physical dynamical systems (Jonschkowski & Brock, 2015) and that use interaction to build up dynamics-aware features (Bengio et al., 2017; Laversanne-Finot et al., 2018).”
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning (2018, unknown): “The idea of autonomously learning modular goal representations from experience has been explored in Laversanne-Finot et al. (2018), using β−V AEs.”
Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards (2019, background): “Other ideas following contingency awareness include controllable feature learning [11, 20], tool use discovery [10] and sensing guidance [17, 18, 19].”
Unsupervised Control Through Non-Parametric Discriminative Rewards (2018, background): “P ´ er ´ e et al. (2018) and Laversanne-Finot et al. (2018) employ a two-stage framework wherein unsupervised representation learning is used to learn a model of the observations from which to sample goals for control in simple simulated environments.”