Goal-directed decision making in prefrontal cortex: a computational framework

By Matthew Botvinick, James An. 2008

56 citations recorded by the source corpus.

Research in animal learning and behavioral neuroscience has distinguished between two forms of action control: a habit-based form, which relies on stored actio n values, and a goal-dir ected form, which forecasts and compares action outcomes based on a model of the environment. While habit-based control has been the subject of extensive computational research, the computational principles underlying goal-directed control in animals have so far received less attention. In the present paper, we advance a computational framework for goal-directed control in animals and humans. We take three empirically motivated points as founding premises: (1) Neurons in dorsolateral prefrontal cortex represent action policies, (2) Neurons in orbitofrontal cortex represent rewards, and (3) Neural computation, across domains, can be appropriately understood as performing structured probabilistic inference. On a purely computational level, the resulting account relates closely to previous work using Bayesian inference to solve Markov decision problems, but extends this work by introducing a new algorithm, which provably converges on optimal plans. On a cognitive and neuroscientific level, the theory provides a unifying framework for several different forms of goal-directed action selection, placing emphasis on a novel form, within which orbitofrontal reward representations directly drive policy selection.

Subfields: Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Neuroscience

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “A number of prior works have also sought to incorporate probabilistic inference into a model of biological decision making and control (Solway and Botvinick, 2012; Botvinick and An, 2009; Botvinick and Toussaint, 2012; Friston, 2009).”
Perception, Action, and Utility: The Tangled Skein (2012, background): “This is a common problem in statistics and machine learning, where the approximations tend to fall into one of two classes: • Monte Carlo approximations (Robert and Casella, 2004), where the posterior is approximated by a set of samples s1:M .”
Active Inference: A Process Theory (2017, background): “Crucially, this scheme entails reciprocal interactions between the prefrontal cortex and basal ganglia (Botvinick & An, 2009), in particular, selection of expected motor outcomes by the basal ganglia (Mannella & Baldassarre, 2015).”
Active Inference, Curiosity and Insight (2017, background): “…literature in these fields or provide a synthesis of established ideas (e.g., Schmidhuber, 1991; Oaksford & Chater, 2001; Koechlin et al., 2003; Botvinick & An, 2008; Nelson et al., 2010; Navarro & Perfors, 2011; Tenenbaum, Kemp, Griffiths, & Goodman, 2011; Botvinick & Toussaint, 2012; Collins…”
Active inference and epistemic value (2015, unknown): “…outcomes and their epistemic (intrinsic) value in reducing uncertainty about environmental contingencies (Bialek, Nemenman, & Tishby, 2001; Botvinick & An, 2008; Braun, Ortega, Theodorou, & Schaal, 2011; Bromberg-Martin & Hikosaka 2009; Cohen, McClure, Correspondence should be addressed…”