Variational Intrinsic Control

By Karol Gregor, Danilo Jimenez Rezende, Daan Wierstra. 2016

484 citations recorded by the source corpus.

In this paper we introduce a new unsupervised reinforcement learning method for discovering the set of intrinsic options available to an agent. This set is learned by maximizing the number of different states an agent can reliably reach, as measured by the mutual information between the set of options and option termination states. To this end, we instantiate two policy gradient based algorithms, one that creates an explicit embedding space of options and one that represents options implicitly. The algorithms also provide an explicit measure of empowerment in a given state that can be used by an empowerment maximizing agent. The algorithm scales well with function approximation and we demonstrate the applicability of the algorithm on a range of tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, background): “The intrinsic motivation measure may include mutual information between actions and end states (Gregor et al., 2016), state prediction error or surprise (Pathak et al.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, methodology): “opose improving data efﬁciency of RL algorithms using self-supervised prediction based auxiliary tasks (Jaderberg et al.,2017;Shelhamer et al.,2017). Fu et al. (2017) learn discriminative models, and Gregor et al. (2017) use empowerment based measure to tackle exploration in sparse reward setups. 6.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “(Gregor et al., 2016; Hausman et al., 2018; Eysenbach et al., 2018) leverage principles of empowerment and discover goals based on information theoretic objectives, learning to represent the intrinsic control space of an agent in order to facilitate learning about more than a single task.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “One objective for achieving this proposed in prior work [20, 14, 40, 34] is the mutual information (MI) between skills and the states they visit: I(s; z) = H(s) − H(s|z).”
Exploration by Random Network Distillation (2018, methodology): “Other methods of exploration include adversarial self-play (Sukhbaatar et al., 2018), maximizing empowerment (Gregor et al., 2017), parameter noise (Plappert et al., 2017; Fortunato et al., 2017), identifying diverse policies (Eysenbach et al., 2018; Achiam et al., 2018), and using ensembles of…”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Warde-Farley et al., 2018; Shankar & Gupta, 2020; Thomas et al., 2018; Sharma et al., 2020).”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “Our work draws inspiration from variational option discovery [1, 9, 13], which— in formal analogy with variational auto-encoders [15]—trains a maximum-entropy policy encoder to map latent context vectors into trajectories from which the context can be recovered by a supervised decoder.”
MAVEN: Multi-Agent Variational Exploration (2019, background): “Similarly [14, 8] learn a mixture of diverse behaviours using deep neural networks to extract state features and use MI maximisation between them and the behaviours to learn useful skills without a reward function.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, methodology): “Three important distinctions allow us to apply our method to tasks signiﬁcantly more complex than the gridworlds in Gregor et al. (2016).”
Disentangling the independently controllable factors of variation by interacting with the world (2018, background): “An alternate idea is that these representations do not need to be learned explicitly, and that learning can be guided through internal mechanisms of reward, usually called intrinsic motivation [Barto et al., Oudeyer and Kaplan, 2009, Salge et al., 2013, Gregor et al., 2017].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “Recently, multiple works (Eysenbach et al., 2018; Gregor et al., 2016; Achiam et al., 2018) apply this idea to learn diverse skills which maximally cover the state space.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “…by imposing information theoretic objectives on learning that incentivize increased option diversity and increased entropy in option selection [Gregor et al., 2016, Florensa et al., 2017, Hausman et al., 2018, Eysenbach et al., 2018, 1A review of this long and at times acrimonious debate is…”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “In Gregor et al. [10], they optimize a quantity called empowerment which is a measurement of the control an agent has over the state.”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, background): “Gregor et al. (2016) aims to learn a maximally discriminative set of options by maximizing the mutual information between the ﬁnal state reached by each of the options and the latent representation.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “This formulation bears interesting similarities with diversity inducing regularization schemes based on mutual information (e.g. Gregor et al., 2017; Florensa et al., 2017) but arises here as an approximation to trajectory entropy.”
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018, background): “A line of work [9, 5, 4, 7, 3] takes this approach in the context of intrinsic motivation.”
Hindsight Credit Assignment (2019, methodology): “However, in our work, the inverse model serves as a way of determining the influence of an action on a future outcome, whereas the work in [6, 8] aims to use the inverse model to derive an intrinsic reward for training policies in which actions influence the future observations.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “In particular, to measure performance here we consider how LiSP maximizes empowerment (Klyubin et al., 2005; Salge et al., 2014; Gregor et al., 2016), which roughly corresponds to how stable the agent is, or how much “potential” it has.”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “…f : D φ → π information objectives can be formulated, including mutual information between single states and z (Eysenbach et al., 2018), pairs of start and end states and z (Gregor et al., 2016), and entire trajectories and z (Achiam et al., 2018; Sharma et al., 2019; Warde-Farley et al., 2018).”