Diversity is All You Need: Learning Skills without a Reward Function

By Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine. 2018

1,305 citations recorded by the source corpus.

Intelligent creatures can explore their environments and learn useful skills without supervision. In this paper, we propose DIAYN ("Diversity is All You Need"), a method for learning useful skills without a reward function. Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy. On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping. In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward. In these environments, some of the learned skills correspond to solving the task, and each skill that solves the task does so in a distinct manner. Our results suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efficiency in reinforcement learning

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, background): “…2016), state prediction error or surprise (Pathak et al., 2017), state visit counts (Storck et al., 1995; Bellemare et al., 2016), distinguishability to a decoder (Eysenbach et al., 2018), or empowerment (Salge et al., 2014), which is also related to mutual information (Mohamed and Rezende, 2015).”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Yang et al. (2019) use DIAYN (Eysenbach et al., 2018) to learn lower-level skills that a higher level QMIX agent uses.”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Others have shown benefits of intrinsic motivation for learning RL tasks (Eysenbach et al., 2018; Kulkarni et al., 2016; Pathak et al., 2017), though the connections to policy robustness and object-based representations are not explored in these works.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “(Gregor et al., 2016; Hausman et al., 2018; Eysenbach et al., 2018) leverage principles of empowerment and discover goals based on information theoretic objectives, learning to represent the intrinsic control space of an agent in order to facilitate learning about more than a single task.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “One objective for achieving this proposed in prior work [20, 14, 40, 34] is the mutual information (MI) between skills and the states they visit: I(s; z) = H(s) − H(s|z).”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “As we show in our experiments, this bi-level initialization is signiﬁcantly more amenable to RRF than learning the high level from scratch as described in [7, 15, 22], and allows us to avoid the expensive goal labeling that is required in [27].”
Exploration by Random Network Distillation (2018, methodology): “…et al., 2018), maximizing empowerment (Gregor et al., 2017), parameter noise (Plappert et al., 2017; Fortunato et al., 2017), identifying diverse policies (Eysenbach et al., 2018; Achiam et al., 2018), and using ensembles of value functions (Osband et al., 2018; 2016; Chen et al., 2017).”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “There are a number of recent works (Eysenbach et al., 2018; Nachum et al., 2018a; Sharma et al., 2019) which use unsupervised objectives to discover skills and use the discovered skills for planning (Sharma et al., 2019), few-shot imitation learning, or online RL (Eysenbach et al., 2018; Nachum et…”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Existing methods such as curiosity-driven algorithms [41–44] or maximising mutual information [45–47] have shown great promise.”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “We also consider an additional baseline, DIAYN (Eysenbach et al., 2019), which discovers diverse skills using a trajectory-based diversity reward.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “A skill is a policy that is conditioned on a latent variable, executed for an extended duration, and generates behavior from which the latent variable can be decoded [1, 9].”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “Hence, agents jointly learn to solve the task is many different ways; this is how MAVEN prevents suboptimality from representational constraints, whereas DIAYN is concerned only with discovering new states.”
Visual Reinforcement Learning with Imagined Goals (2018, methodology): “could be used for this stage, including off-the-shelf exploration bonuses [29, 4] or unsupervised reinforcement learning methods [9, 12].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “Recently, multiple works (Eysenbach et al., 2018; Gregor et al., 2016; Achiam et al., 2018) apply this idea to learn diverse skills which maximally cover the state space.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “…objectives on learning that incentivize increased option diversity and increased entropy in option selection [Gregor et al., 2016, Florensa et al., 2017, Hausman et al., 2018, Eysenbach et al., 2018, 1A review of this long and at times acrimonious debate is outside of the scope of this paper.”
Large-Scale Study of Curiosity-Driven Learning (2018, methodology): “In a concurrent work, diversity is used as a measure to learn skills without reward functions Eysenbach et al. [7].”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, background): “The results then become the foundation for imitation learning and hierarchical RL [43].”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, background): “, 2017), or constructing diversityseeking priors that cause lower-layer primitives to take on different roles (Daniel et al., 2012; Eysenbach et al., 2018).”