Variational Option Discovery Algorithms

By Joshua Achiam, Harrison Edwards, Dario Amodei, Pieter Abbeel. 2018

209 citations recorded by the source corpus.

We explore methods for option discovery based on variational inference and make two algorithmic contributions. First: we highlight a tight connection between variational option discovery methods and variational autoencoders, and introduce Variational Autoencoding Learning of Options by Reinforcement (VALOR), a new method derived from the connection. In VALOR, the policy encodes contexts from a noise distribution into trajectories, and the decoder recovers the contexts from the complete trajectories. Second: we propose a curriculum learning approach where the number of contexts seen by the agent increases whenever the agent's performance is strong enough (as measured by the decoder) on the current set of contexts. We show that this simple trick stabilizes training for VALOR and prior variational option discovery methods, allowing a single agent to learn many more modes of behavior than it could with a fixed context distribution. Finally, we investigate other topics related to variational option discovery, including fundamental limitations of the general approach and the applicability of learned options to downstream tasks.

Subfields: Artificial Intelligence, Management Science and Operations Research, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Ocean Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Achiam et al. (2018) relate variational discovery of options to variational auto-encoders and pose it as an optimization problem.”
Exploration by Random Network Distillation (2018, methodology): “…et al., 2018), maximizing empowerment (Gregor et al., 2017), parameter noise (Plappert et al., 2017; Fortunato et al., 2017), identifying diverse policies (Eysenbach et al., 2018; Achiam et al., 2018), and using ensembles of value functions (Osband et al., 2018; 2016; Chen et al., 2017).”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “variables z , each of which corresponds to a skill . to generalize Z to be a learned continuous embedding space [1].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “Recently, multiple works (Eysenbach et al., 2018; Gregor et al., 2016; Achiam et al., 2018) apply this idea to learn diverse skills which maximally cover the state space.”
Learning Novel Policies For Tasks (2019, methodology): “It may be tempting to consider using option discovery approaches such as DIAYN (Eysenbach et al., 2018) or VALOR (Achiam et al., 2018) to learn novel policies.”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, methodology): “Unsupervised pre-training has been studied for extracting strong behavioral priors that can be utilized to solve downstream tasks efﬁciently in the context of RL (Daniel et al., 2016; Florensa et al., 2018; Achiam et al., 2018; Eysenbach et al., 2019; Sharma et al., 2020).”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “…f : D φ → π information objectives can be formulated, including mutual information between single states and z (Eysenbach et al., 2018), pairs of start and end states and z (Gregor et al., 2016), and entire trajectories and z (Achiam et al., 2018; Sharma et al., 2019; Warde-Farley et al., 2018).”
Efficient Exploration via State Marginal Matching (2019, methodology): “We further decompose the SMM objective into a mixture of distributions, and derive an algorithm for learning a mixture of policies that resembles the mutual-information objectives in recent work (Achiam et al., 2018; Eysenbach et al., 2018; Co-Reyes et al., 2018).”
Multitask Soft Option Learning (2019, background): “Recently, several authors have proposed unsupervised training objectives for learning diverse skills based on their distinctiveness (Gregor et al., 2016; Florensa et al., 2017; Achiam et al., 2018; Eysenbach et al., 2019).”
Deep Hierarchical Planning from Pixels (2022, methodology): “Several variations of this approach have been developed with further improvements in stability and diversity of skills, including SSN4HRL (Florensa et al., 2017), DIAYN (Eysenbach et al., 2018) and VALOR (Achiam et al., 2018).”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, methodology): “9 we recover the information maximization objective used in prior work [1, 16], besides the fact that we simulatenously perform meta-learning.”
Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction (2019, unknown): “[1] discuss a connection between variational option discovery and variational auto-encoders by solving a max-max optimization problem.”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “We learn latent-conditioned policies π ( a | s, z ) , and deﬁne skills or options as the policies obtained when conditioning π on a ﬁxed value of z ∈ Z (Eysenbach et al., 2019; Achiam et al., 2018).”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Most other works (Achiam et al., 2018; Warde-Farley et al., 2018; Hansen et al., 2019; Campos et al., 2020) are complementary methods that can help learn a set of skills.”
Action and Perception as Divergence Minimization (2020, methodology): “(21) VIC (Gregor et al., 2016) introduced information-based skill discovery as an extension of empowerment, motivating a line of work including SNN (Florensa et al., 2017), DIAYN (Eysenbach et al., 2018), work by Hausman et al. (2018), VALOR (Achiam et al., 2018), and work by Tirumala et al. (2019) ”
Agent57: Outperforming the Atari Human Benchmark (2020, background): “, 2018), unsupervised policy learning (Gregor et al., 2016; Achiam et al., 2018; Eysenbach et al., 2018) and intrinsic motivation (Schmidhuber, 1991; Oudeyer et al.”