Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

By Shakir Mohamed, Danilo Jimenez Rezende. 2015

443 citations recorded by the source corpus.

The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm --- an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, background): “, 2014), which is also related to mutual information (Mohamed and Rezende, 2015).”
Concrete Problems in AI Safety (2016, background): “In the real world, however, tasks often involve bringing the external world into some objective state, which the agent can only ever confirm through imperfect perceptions.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, background): “Mohamed and Rezende (2015) also considered a variational approach to the different problem of maximizing an agent’s ability to inﬂuence its environment.”
Deep Variational Information Bottleneck (2017, background): “Mohamed & Rezende (2015) also explore variational bounds on mutual information, and apply them to deep neural networks, but in the context of reinforcement learning.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “…the agent to perform actions that reduce the error/uncertainty in the agent’s ability to predict the consequence of its own actions (i.e. its knowledge about the environment) (Houthooft et al., 2016; Mohamed & Rezende, 2015; Schmidhuber, 1991; 2010; Singh et al., 2005; Stadie et al., 2015).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Similarly, other work has instead used the empowerment maximization principle [38] to define mutual information based objectives from which to learn behavioral primitives [30, 20, 16, 14, 40].”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Another paradigm are transition-based methods, in which agents are rewarded for high prediction error in a learned forward or inverse dynamics model [42, 43, 44, 45, 46, 47, 48, 49].”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Mohamed & Rezende (2015) and Jung et al. (2011) use the mutual information between states and actions as a notion of empowerment for an intrinsically motivated agent.”
Variational Intrinsic Control (2016, background): “Classical empowerment (Salge et al., 2014; Mohamed & Rezende, 2015) maximizes mutual information between sequences of actions A = a 1 , . . . , a T and ﬁnal states.”
Deep Successor Reinforcement Learning (2016, methodology): “In addition to subgoals, using DSR for extracting other intrinsic motivation measures such as improvements to the predictive world model [30] or mutual information [23] is worth pursuing.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “Examples of such works include empowerment [37, 47], count-based exploration [8, 50, 67, 21], information gain about agent’s dynamics [64] and forward-inverse dynamics models [53].”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “model prediction error [112, 86] or maximising information gain [73, 42].”
Large-Scale Study of Curiosity-Driven Learning (2018, background): “Examples of intrinsic reward include “curiosity” [11, 22, 27, 35, 40] which uses prediction error as reward signal, and “visitation counts” [3, 20, 24, 30] which discourages the agent from revisiting the same states.”
Learning Awareness Models (2018, background): “…Oudeyer & Kaplan (2008; 2009); Sequeira et al. (2011); Still & Precup (2012); Bellemare et al. (2016); Martius et al. (2013); Schmidhuber (2008); Mohamed & Rezende (2015); Haber et al. (2018b;a) Some of the ideas here, in particular the notion of choosing actions speciﬁcally to improve a model…”
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning (2017, methodology): “Mohamed and Rezende [14] developed a scalable method of approximating empowerment, the mutual information between an agent’s actions and the future state of the environment, using variational methods.”