Self-Supervised Exploration via Disagreement

By Deepak Pathak, Dhiraj Gandhi, Abhinav Kumar Gupta. 2019

460 citations recorded by the source corpus.

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at this https URL

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “This focuses on intrinsic rewards and motivation rather than external rewards from the environment, which is beneficial in cases where external rewards are sparse [65].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “This includes demonstration data, data from modern exploration methods that scale to pixel observations [51, 52, 53], and task-driven data from running reinforcement learning on particular tasks.”
Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation (2020, methodology): “To aid EE-Space exploration, we evaluated the approach from [40], which gives the agent additional exploration rewards.”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “…focused on encouraging novelty-seeking behaviors by maximizing model uncertainty (Houthooft et al., 2016; Still & Precup, 2012; Shyam et al., 2018; Pathak et al., 2019), by maximizing model prediction error or improvement (Lopes et al., 2012; Pathak et al., 2017), through state visitation counts…”
Unity: A General Platform for Intelligent Agents (2018, methodology): “Unity environments have been used in varied research such as intrinsic motivation (Burda et al., 2019; Pathak et al., 2019a), Neural Attention (Ghani et al., 2018), and semi-parametric reinforcement learning (Jain and Lindsey, 2018).”
Semantic Curiosity for Active Visual Learning (2020, unknown): “This is in a departure from existing works that measure it in 2D image space [37], or consistency among multiple models [38].”
Ready Policy One: World Building Through Active Learning (2020, methodology): “It is also possible to maximize information gain for next state predictions (as opposed to rewards), and this is similar to the approach in [40].”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “Another approximation could be to use ensembles of models as proposed in [45].”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, background): “Similarly, in the MAL formulation, entropy regularization [51, 23] and disagreement based reward bonuses [22, 52] lead to adversarial best response by encouraging the policy to visit parts of the state space where the model is likely to be inaccurate.”
Meta-learning curiosity algorithms (2020, background): “, 2018), and ensemble predictive variance (Pathak et al., 2019), are shown in figure 2 (bigger versions can be found in appendix A.”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Intrinsic rewards can be derived from various measures: novelty [43, 55, 6, 56], prediction-error [38, 3], ensemble disagreement [11, 39, 48, 46, 18, 50], or information gain [23].”
Intrinsic Motivation for Encouraging Synergistic Behavior (2020, background): “…actual value of some function computed on the current state, the taken action, etc. (Barto, 2013; Oudeyer et al., 2007; Bellemare et al., 2016); intrinsic motivation can even be used on its own when no extrinsic reward is provided (Pathak et al., 2017; 2019; Burda et al., 2019; Haber et al., 2018).”
An Exploration of Embodied Visual Exploration (2020, unknown): “Further variants of curiosity aim to overcome the “noisy-TV” problem where the curiosity agent fixates on parts of the environment where stochasticity is a function of its actions (Burda et al. 2018a, b; Pathak et al. 2019).”
Flexible Option Learning (2021, background): “A possible remedy would be to favor diversity, or similar measures, through reward shaping [Gregor et al., 2016, Eysenbach et al., 2018, Pathak et al., 2019, Klissarov and Precup, 2020].”
Planning to Explore via Self-Supervised World Models (2020, background): “Similar to our work, other recent approaches use the notion of model disagreement to encourage visiting states with the highest potential to improve the model (Burda et al., 2018; Pathak et al., 2019), motivated by the active learning literature (Seung et al.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “Unsupervised pretraining or self-supervised learning using world models (Ha and Schmid-huber, 2018; Shyam et al., 2018; Pathak et al., 2019; Sharma et al., 2020; Sekar et al., 2020) push against the tabula rasa viewpoint of agents starting from zero knowledge about the environment it is interacting…”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “Other works also propose to use ensemble of networks for measuring uncertainty (Pathak et al., 2019; Shyam et al., 2019).”