Empowerment - an Introduction

By Christoph Salge, Cornelius Glackin, Daniel Polani. 2013

215 citations recorded by the source corpus.

Is it better for you to own a corkscrew or not? If asked, you as a human being would likely say “yes”, but more importantly, you are somehow able to make this decision. You are able to decide this, even if your current acute problems or task do not include opening a wine bottle. Similarly, it is also unlikely that you evaluated several possible trajectories your life could take and looked at them with and without a corkscrew, and then measured your survival or reproductive fitness in each. When you, as a human cognitive agent, made this decision, you were likely relying on a behavioural “proxy”, an internal motivation that abstracts the problem of evaluating a decision impact on your overall life, but evaluating it in regard to some simple fitness function. One example would be the idea of curiosity, urging you to act so that your experience new sensations and learn about the environment. On average, this should lead to better and richer models of the world, which give you a better chance of reaching your ultimate goals of survival and reproduction.

Subfields: Cognitive Neuroscience, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, background): “…2016), state prediction error or surprise (Pathak et al., 2017), state visit counts (Storck et al., 1995; Bellemare et al., 2016), distinguishability to a decoder (Eysenbach et al., 2018), or empowerment (Salge et al., 2014), which is also related to mutual information (Mohamed and Rezende, 2015).”
Concrete Problems in AI Safety (2016, background): “Perhaps the bestknown such measure is empowerment [129], the maximum possible mutual information between the agent’s potential future actions and its potential future state (or equivalently, the Shannon capacity of the channel between the agent’s actions and the environment).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Similarly, other work has instead used the empowerment maximization principle [38] to define mutual information based objectives from which to learn behavioral primitives [30, 20, 16, 14, 40].”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, methodology): “…on prediction errors of future states such predicted L 1 change, predicted mode change or probability gain [16], or salient event prediction [23]; and measures based on information-theoretic quantities such as predicted information gain (PIG) [14], causal entropic forces [28] or empowerment [21].”
Variational Intrinsic Control (2016, methodology): “This information measure has been introduced in the empowerment literature before (Salge et al., 2014; Klyubin et al., 2005) along with methods for measuring it (such as Blahut (1972); Arimoto (1972)).”
Disentangling the independently controllable factors of variation by interacting with the world (2018, neutral): “An alternate idea is that these representations do not need to be learned explicitly, and that learning can be guided through internal mechanisms of reward, usually called intrinsic motivation [Barto et al., Oudeyer and Kaplan, 2009, Salge et al., 2013, Gregor et al., 2017].”
AI Safety Gridworlds (2017, methodology): “This could be done using an information-theoretic measure such as empowerment (Salge et al., 2014), the maximum mutual information between the agent’s actions and its future state.”
Avoiding Side Effects By Considering Future Tasks (2020, background): “The future task approach is related to empowerment [9, 18], a measure of the agent’s control over its environment.”
Penalizing Side Effects using Stepwise Relative Reachability (2018, unknown): “Our RR measure is related to empowerment [Klyubin et al., 2005; Salge et al., 2014; Gregor et al., 2017], a measure of the agent’s control over its environment, defined as the highest possible mutual information between the agent’s actions and the future state.”
Optimal Policies Tend To Seek Power (2019, methodology): “Appendix A demonstrates that our formalization returns intuitive verdicts in situations where information-theoretic empowerment does not [Salge et al., 2014].”
A Laplacian Framework for Option Discovery in Reinforcement Learning (2017, background): “Recently, Gregor et al. (2016) proposed an algorithm in which agents discover options by maximizing a notion of empowerment (Salge et al., 2014), where the agent aims at getting to states with a maximal set of available intrinsic options.”
Unsupervised Real-Time Control Through Variational Empowerment (2017, unknown): “In particular, we opt for empowerment [Klyubin et al., 2005, Salge et al., 2014], an information-theoretic formulation of the agent’s inﬂuence on the near future.”
A deep learning framework for neuroscience (2019, background): “We will return to some of the specific objective functions proposed for the brain below 50–53 .”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, background): “A large body of prior work has studied exploration and intrinsic motivation objectives [51, 48, 43, 22, 8, 5, 35, 42].”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “We can also reward the agent by Empowerment (Klyubin et al., 2005; Gregor et al., 2016; Salge et al., 2014; Mohamed and Rezende, 2015), prioritizing the states that agent can take control through its actions.”
Measuring and avoiding side effects using relative reachability (2018, background): “Our relative reachability measure is related to empowerment (Klyubin et al., 2005; Salge et al., 2014; Mohamed and Rezende, 2015; Gregor et al., 2017), a measure of the agent’s control over its environment, defined as the highest possible mutual information between the agent’s actions and the future”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, background): “The idea of unsupervised RL is often formulated through the lens of empowerment (Salge et al., 2014), which formalizes the notion of an agent discovering what can be done in an environment while learning how to do it.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “In particular, to measure performance here we consider how LiSP maximizes empowerment (Klyubin et al., 2005; Salge et al., 2014; Gregor et al., 2016), which roughly corresponds to how stable the agent is, or how much “potential” it has.”
What can I do here? A Theory of Affordances in Reinforcement Learning (2020, background): “This notion of intent is similar to empowerment (Salge et al., 2014) or to setting a goal for the agent to reach after a temporally extended action (Schaul et al., 2015; Nachum et al., 2018).”