Action and Perception as Divergence Minimization

By Danijar Hafner, Pedro A. Ortega, Jimmy Ba, Thomas Parr, Karl J. Friston, Nicolas Manfred Otto Heess. 2020

60 citations recorded by the source corpus.

We introduce a unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions to align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's representations and inputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as a latent variable model suggests powerful world models as a path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common language for comparing a wide range of objectives, advances the understanding of latent variables for decision making, and offers a recipe for designing novel objectives. We recommend deriving future agent objectives the joint divergence to facilitate comparison, to point out the agent's target distribution, and to identify the intrinsic objective terms needed to reach that distribution.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Information Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Information Theory

Citations in the corpus

On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “The idea of posing control problems as distribution matching has resurfaced numerous times in the RL literature (Kappen et al., 2012; Friston et al., 2010; Levine, 2018; Hafner et al., 2020; Buckley et al., 2017).”
RL with KL penalties is better viewed as Bayesian inference (2022, methodology): “…there are frameworks such as active inference (Friston et al., 2010; Buckley et al., 2017) or action and perception as divergence minimisation (Hafner et al., 2020) that further generalise control as inference to a principle of minimising the KL divergence from a probability distribution…”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, background): “Especially in the continuous control domain, interpreting RL as probabilistic inference [60, 61] has yielded many kinds of algorithms with strong empirical performances [34, 50, 51, 12, 29, 21, 16, 24].”
Deep Hierarchical Planning from Pixels (2022, background): “…Hausman et al., 2018; Eysenbach et al., 2018; Achiam et al., 2018; Merel et al., 2018; Laskin et al., 2022; Sharma et al., 2019; Xie et al., 2020; Hafner et al., 2020b; Strouse et al., 2021), or training the low-level as a goal-conditioned policy (Andrychowicz et al., 2017; Levy et al., 2017;…”
Reward is enough (2021, background): “The reinforcement learning problem may also be transformed into a probabilistic framework that approximates the goal of reward maximisation [66,39,26,17].”
The free energy principle made simpler but not too simple (2022, unknown): “Please see [148,150] for a treatment of things like empowerment [170], information bottleneck [171] and predictive information [172,173].”