Reinforcement Learning or Active Inference?

By Karl J. Friston, Jean Daunizeau, Stefan J. Kiebel. PLoS ONE, 2009

463 citations recorded by the source corpus.

This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sampling of the environment to minimize their free-energy. Such agents learn causal structure in the environment and sample it in an adaptive and self-supervised fashion. This results in behavioural policies that reproduce those optimised by reinforcement learning and dynamic programming. Critically, we do not need to invoke the notion of reward, value or utility. We illustrate these points by solving a benchmark problem in dynamic programming; namely the mountain-car problem, using active perception or inference under the free-energy principle. The ensuing proof-of-concept may be important because the free-energy formulation furnishes a unified account of both action and perception and may speak to a reappraisal of the role of dopamine in the brain.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/5e07da2914783a21980ecc1cea688d1333e6b6e4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Perception, Action, and Utility: The Tangled Skein (2012, background): “Friston and colleagues (Friston et al., 2009, 2010) formulate the decision optimization problem in these terms.”
The free energy principle made simpler but not too simple (2022, unknown): “3; see [5,57–59] for some formal accounts of these relationships.”
Action and behavior: a free-energy formulation (2010, methodology): “This entails adjusting the parameters to maximize the ensemble density, p(˜ x|m) at the target location, as described in Friston et al. (2009) .”
Optimal control as a graphical model inference problem (2009, unknown): “The class of KL control problems contains interesting special cases such as the continuous non-linear Gaussian stochastic control problems introduced in Kappen (2005), discrete planning tasks and multi-agent games, as illustrated in this paper.”
Affective Value in the Predictive Mind (2017, background): “In fact, within this view, there are no distinct reward or punishment stimuli (Friston et al. 2009; Wörgötter and Porr 2005).”
Action understanding and active inference (2011, unknown): “Furthermore , it is exactly the same scheme used in previous illustrations of perceptual inference (Kiebel et al. 2009a), perceptual learning (Friston 2008), reinforcement learning (Friston et al. 2009), active inference (Friston et al. 2010a) and attentional processing (Feldman and Friston 2010).”
Free Energy, Value, and Attractors (2011, methodology): “The free energy formulation uses ideas from statistical physics to explain action in terms of minimising sensory surprise.”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, methodology): “Inspired by the free energy principle (Friston, 2009; Friston et al., 2009; Ueltzhöffer, 2018; Faraji et al., 2018; Friston et al., 2016) including recent methods that train policies using RL (Tschantz et al.”
Hierarchical Active Inference: A Theory of Motivated Control (2018, methodology): “Technically, this uses Bayesian model selection based on the expected free energy [41].”
The role of prediction in social neuroscience (2012, methodology): “Inference can refer to deterministic short-term processes that are largely situated in current behavior and are probabilistic estimations about the state of the world, and are most relevant to prediction errors and concepts modeled with Bayesian statistics (Friston et al., 2009).”
The Computational Anatomy of Psychosis (2013, background): “In short, active inference can be regarded www.frontiersin.org May 2013 | Volume 4 | Article 47 | 9 as equipping a generalized predictive coding scheme with classical reflex arcs: see Friston et al. (2009) and Adams et al. (2013) for details.”