Empowerment for continuous agent—environment systems

By Tobias Jung, Daniel Polani, Peter Stone. 2011

110 citations recorded by the source corpus.

This article develops generalizations of empowerment to continuous states. Empowerment is a recently introduced information-theoretic quantity motivated by hypotheses about the efficiency of the sensorimotor loop in biological organisms, but also from considerations stemming from curiosity-driven learning. Empowerment measures, for agent—environment systems with stochastic transitions, how much influence an agent has on its environment, but only that influence that can be sensed by the agent sensors. It is an information-theoretic generalization of joint controllability (influence on environment) and observability (measurement by sensors) of the environment by the agent, both controllability and observability being usually defined in control theory as the dimensionality of the control/observation spaces. Earlier work has shown that empowerment has various interesting and relevant properties, for example, it allows us to identify salient states using only the dynamics, and it can act as intrinsic reward without requiring an external reward. However, in this previous work empowerment was limited to the case of small-scale and discrete domains and furthermore state transition probabilities were assumed to be known. The goal of this article is to extend empowerment to the significantly more important and relevant case of continuous vector-valued state spaces and initially unknown state transition probabilities. The continuous state space is addressed by Monte Carlo approximation; the unknown transitions are addressed by model learning and prediction for which we apply Gaussian processes regression with iterated forecasting. In a number of well-known continuous control tasks we examine the dynamics induced by empowerment and include an application to exploration and online model learning.

Subfields: Cognitive Neuroscience, Global and Planetary Change, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Global and Planetary Change

In Molecular Biology

Citations in the corpus

Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Mohamed & Rezende (2015) and Jung et al. (2011) use the mutual information between states and actions as a notion of empowerment for an intrinsically motivated agent.”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, background): “In the continuous domain, Monte Carlo integration has been proposed [10], but applications of Monte Carlo estimators can require a large number of draws to obtain accurate solutions and manageable variance.”
Empowerment - an Introduction (2013, methodology): “Another approximation to compute empowerment which can still deal with any kind of p(s|a) is Monte Carlo Integration (Jung et al. 2011).”
The alignment problem from a deep learning perspective (2022, background): “However, this will become signiﬁ-cantly more difﬁcult as policies develop situational awareness , as described below.”
Unsupervised Real-Time Control Through Variational Empowerment (2017, unknown): “In particular, Jung et al. [2011] discretise the action space; Salge et al. [2013] linearise pendulum dynamics to make empowerment tractable.”