Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates.

By Alec Solway, Matthew Botvinick. 2012

198 citations recorded by the source corpus.

Recent work has given rise to the view that reward-based decision making is governed by two key controllers: a habit system, which stores stimulus-response associations shaped by past reward, and a goal-oriented system that selects actions based on their anticipated outcomes. The current literature provides a rich body of computational theory addressing habit formation, centering on temporal-difference learning mechanisms. Less progress has been made toward formalizing the processes involved in goal-directed decision making. We draw on recent work in cognitive neuroscience, animal conditioning, cognitive and developmental psychology and machine learning, to outline a new theory of goal-directed decision making. Our basic proposal is that the brain, within an identifiable network of cortical and subcortical structures, implements a probabilistic generative model of reward, and that goal-directed decision making is effected through Bayesian inversion of this model. We present a set of simulations implementing the account, which address benchmark behavioral and neuroscientific findings, and which give rise to a set of testable predictions. We also discuss the relationship between the proposed framework and other models of decision making, including recent models of perceptual choice, to which our theory bears a direct connection.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “A number of prior works have also sought to incorporate probabilistic inference into a model of biological decision making and control (Solway and Botvinick, 2012; Botvinick and An, 2009; Botvinick and Toussaint, 2012; Friston, 2009).”
Compress and Control (2014, background): “These ideas have been recently explored in both the neuroscience (Botvinick and Toussaint 2012; Solway and Botvinick 2012) and machine learning (Attias 2003; Poupart, Lang, and Toussaint 2011) literature.”
Active Inference: A Process Theory (2017, methodology): “…and prefrontal cortex, such as evidence accumulation and race-to-bound or threshold (Huk & Shadlen, 2005, Gold & Shadlen, 2007; Hunt et al., 2012; Solway & Botvinick, 2012; de Lafuente, Jazayeri, & Shadlen, 2015; FitzGerald, Moran, Friston, & Dolan, 2015; Latimer, Yates, Meister, Huk, & Pillow,…”
Model-based hierarchical reinforcement learning and human action control (2014, background): “Over the past few years, a series of studies investigating possible roles for modelbased RL in human and animal decision-making has renewed interest in the time-honoured problem of planning [6,7] (see [8]).”
Cognitive Control Predicts Use of Model-based Reinforcement Learning (2014, result): “This points to an altogether different and more detailed process-level account of competition than suggested on previous RL work, which might speak particularly to the within-trial time dynamics of the tradeoff during the choice process (see also Solway & Botvinick, 2012).”
Goals and Habits in the Brain (2013, unknown): “An alternative idea is to transform control-theoretic calculations of the optimal policy into the sort of probabilistic inference problems that are generally believed to be solved by sensory processing regions of the cortex in order to interpret input (Solway and Botvinick, 2012).”
Active Inference, Curiosity and Insight (2017, unknown): “…ideas (e.g., Schmidhuber, 1991; Oaksford & Chater, 2001; Koechlin et al., 2003; Botvinick & An, 2008; Nelson et al., 2010; Navarro & Perfors, 2011; Tenenbaum, Kemp, Griffiths, & Goodman, 2011; Botvinick & Toussaint, 2012; Collins & Koechlin, 2012; Solway & Botvinick, 2012; Donoso et al., 2014).”
Learning to build physical structures better over time (2020, background): “…by recent advances in theories of human planning that highlight the pervasive role of mental simulation in guiding human sequential decision making (Solway & Botvinick, 2015, 2012; Daw, Gershman, Seymour, Dayan, & Dolan, 2011), combined with reasonable assumptions about the cognitive costs of…”
Active inference and epistemic value (2015, unknown): “…Discussion Paper and Commentaries) © 2015 Taylor & Francis & Yu, 2007; Daw, Niv, & Dayan, 2005; Daw, O’Doherty, Dayan, Seymour, & Dolan, 2006; Friston et al., 2014; Pezzulo & Castelfranchi, 2009; Schmidhuber, 1991; Solway & Botvinick, 2012; Still, 2009; Tishby & Polani, 2010).”
The Hierarchical Organisation of Cortical and Basal-Ganglia Systems: A Computationally-Informed Review and Integrated Hypothesis (2013, unknown): “These models are focussed on the learning processes leading to acquire the capacity to select actions by trial-and-error (Table 1), and have been also developed to capture the hierarchical organisation of the brain (Botvinick et al., 2008; Daw et al., 2005; Solway and Botvinick, 2012).”
Affective Value in the Predictive Mind (2017, background): “Specifically, while in classical reinforcement learning goal-directed decision making consists of finding the policy that maximizes expected reward, when framing it in terms of Bayesian inference one assumes reward attainment and finds the policy (state-action pair) that best explains or causes that”