Active inference and epistemic value

By Karl J. Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas H. B. FitzGerald, Giovanni Pezzulo. 2015

705 citations recorded by the source corpus.

We offer a formal treatment of choice behavior based on the premise that agents minimize the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (or intrinsic) value. Minimizing expected free energy is therefore equivalent to maximizing extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximizing information gain or intrinsic value (or reducing uncertainty about the causes of valuable outcomes). The resulting scheme resolves the exploration-exploitation dilemma: Epistemic value is maximized until there is no further information gain, after which exploitation is assured through maximization of extrinsic value. This is formally consistent with the Infomax principle, generalizing formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk-sensitive (Kullback-Leibler) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems, ad hoc softmax parameters become the expected (Bayes-optimal) precision of beliefs about, or confidence in, policies. This article focuses on the basic theory, illustrating the ideas with simulations. A key aspect of these simulations is the similarity between precision updates and dopaminergic discharges observed in conditioning paradigms.

Subfields: Cognitive Neuroscience, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Cognitive Science & Psychology

Citations in the corpus

Continual Lifelong Learning with Neural Networks: A Review (2018, background): “Theuse of generativemodels,which are implicit in predictive coding, is one component embedded in the framework of active inference (Friston et al., 2015).”
How to Entrain Your Evil Demon (2017, background): “The alternative is a model that sees mind as non-representational, extended, situated, embodied or enactive (Hurley 1998; Noë 2004; Gallagher 2005; Thompson 2007; Clark 2008; Hutto and Myin 2013).”
A free energy principle for a particular physics (2019, background): “…inference appears to be sufficient to account for many aspects of behaviour; ranging from the way that we sample our visual world with saccadic eye movements (Mirza et al., 2016), through to how we accumulate evidence for generative models and subsequently make decisions (Friston et al., 2015b).”
Action and Perception as Divergence Minimization (2020, methodology): “Model parameters lead to agents that discover the rules of their environment, such as in active inference (Friston et al., 2015), VIME (Houthooft et al., 2016), MAX (Shyam et al., 2018), and Plan2Explore (Sekar et al., 2020).”
Hierarchical Active Inference: A Theory of Motivated Control (2018, background): “This box explains in more detail the computational study (waiting game) shown in Figure 1, which addresses the importance of withholding prepotent responses [93] (see [94,95] for simulations of exploration–exploitation).”
Navigating the Affordance Landscape: Feedback Control as a Process Model of Behavior and Cognition. (2016, neutral): “This is a potential mechanism through which lower layer actions that are afforded sufficient gain or precision might become routinized and activated automatically by a given situation rather than by top-down signals [77].”
What do we mean by prediction in language comprehension? (2015, background): “…it as defining the generative model that the agent is employing at any given time, such that the goal is achieved by minimising Bayesian surprise across the whole model (see Friston et al., 2015, for a more general discussion of the relationships between utility and generative models).”
Active Inference, Curiosity and Insight (2017, methodology): “, 2013), two-step maze tasks (Friston et al., 2015), evidence accumulation in the urn task (FitzGerald, Schwartenbeck, Moutoussis, Dolan, & Friston, 2015), trust games from behavioral economics (Moutoussis, Trujillo-Barreto, El-Deredy, Dolan, & Friston, 2014), addictive behavior (Schwartenbeck, Fitz”
Active Inference: A Process Theory (2017, methodology): “The simulations use the same setup as in Friston et al. (2015) and is as simple as possible while illustrating some fairly complicated behaviors.”
Affective Value in the Predictive Mind (2017, background): “This implies that action may lead to increase of the distance to a goal (prediction error), in order to move to a familiar position where one can approach the goal with larger certainty (Friston et al. 2015).”