Empowerment: a universal agent-centric measure of control

By Alexander S. Klyubin, Daniel Polani, Chrystopher L. Nehaniv. 2005

429 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “(2020) propose a reverse KL from a joint distribution over observations and latent variables as a universal objective for action and perception that — depending on a choice of the target p — gives rise to many familiar objectives, including empowerment (Klyubin et al., 2005), maximum entropy RL (Haa”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Others have used empowerment, which is the information gain based on entropy of actions, as intrinsic rewards (Klyubin et al., 2005; Mohamed & Rezende, 2015).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Autonomous acquisition of skills is related to the learning with intrinsic motivation, which considers the unsupervised setting where an agent must learn without extrinsic reward [39, 9, 26, 4, 3].”
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play (2017, background): “Another line of work on intrinsic motivation is a formalization of the notion of empowerment (Klyubin et al., 2005), or how much control the agent has over its environment.”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, methodology): “For intrinsic motivation, we use an internal reward measure referred to as empowerment [11, 21] that is obtained by searching for the maximal mutual information I ( · , · ) , conditioned on a starting state s , between a sequence of K actions a and the ﬁnal state reached s (cid:48) : where a = { a…”
Variational Intrinsic Control (2016, methodology): “This information measure has been introduced in the empowerment literature before (Salge et al., 2014; Klyubin et al., 2005) along with methods for measuring it (such as Blahut (1972); Arimoto (1972)).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “Examples of such works include empowerment [37, 47], count-based exploration [8, 50, 67, 21], information gain about agent’s dynamics [64] and forward-inverse dynamics models [53].”
State Representation Learning for Control: An Overview (2018, background): “Related with this prior also is the notion of empowerment [Klyubin et al., 2005], deﬁned as an information-theoretic capacity of an agent’s actuation channel to inﬂuence its own evolution.”
Empowerment - an Introduction (2013, methodology): “Although this n-step horizon provides empowerment with an essentially local “cone of view”, Klyubin et al. (2005a) showed in the maze example that empowerment of a location is negatively correlated with the average distance of that location to all other locations in the maze.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “In particular, to measure performance here we consider how LiSP maximizes empowerment (Klyubin et al., 2005; Salge et al., 2014; Gregor et al., 2016), which roughly corresponds to how stable the agent is, or how much “potential” it has.”
Empowerment for continuous agent—environment systems (2011, background): “…actions of picking up and dropping off a passenger open new degrees of freedom, but only at specific locations in the maze (another example is the ‘‘box pushing’’ scenario where an agent’s empowerment increases close to a pushable box due to the increased number of options; Klyubin et al., 2005a).”
The free energy principle made simpler but not too simple (2022, neutral): “Please see [148,150] for a treatment of things like empowerment [170], information bottleneck [171] and predictive information [172,173].”
Planning to Explore via Self-Supervised World Models (2020, background): “…seek inputs that it cannot yet predict accurately (Schmid-huber, 1991b; Oudeyer et al., 2007; Pathak et al., 2017), maximally inﬂuence its inputs (Klyubin et al., 2005; Eysen-bach et al., 2018), or visit rare states (Poupart et al., 2006; Lehman & Stanley, 2011; Bellemare et al., 2016; Burda et…”
Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) (2010, background): “’s seemingly related approach to intrinsic motivation [36] of 2005 tries to maximize empowerment by maximizing the information an agent could potentially “inject” into its future sensory inputs via a sequence of actions.”
Malthusian Reinforcement Learning (2018, background): “In single-agent reinforcement learning, the main approach is to rely on intrinsic motivations, e.g., for individual curiosity [2, 6, 25, 29, 30, 33], empowerment [22], or social influence [19].”
Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, background): “A number of works have studied the empowerment function in relation to exploration and reinforcement learning (Baumli, Warde-Farley, Hansen, & Mnih, 2020; Jung, Polani, & Stone, 2011; Klyubin et al., 2005; Oudeyer & Kaplan, 2009) as well as in biological systems.”