Decision theory, reinforcement learning, and the brain

By Peter S. Dayan, Nathaniel D. Daw. 2008

600 citations recorded by the source corpus.

Decision making is a core competence for animals and humans acting and surviving in environments they only partially comprehend, gaining rewards and punishments for their troubles. Decision-theoretic concepts permeate experiments and computational models in ethology, psychology, and neuroscience. Here, we review a well-known, coherent Bayesian approach to decision making, showing how it unifies issues in Markovian decision problems, signal detection psychophysics, sequential sampling, and optimal exploration and discuss paradigmatic psychological and neural examples of each problem. We discuss computational issues concerning what subjects know about their task and how ambitious they are in seeking optimal solutions; we address algorithmic topics concerning model-based and model-free methods for making choices; and we highlight key aspects of the neural implementation of decision making.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Go and no-go learning in reward and punishment: Interactions between affect and effect (2012, result): “Future refinement in experimental design might enable detection of just such a segregated system, as indeed has been previously observed (Balleine et al., 2009), including in our laboratories (Dayan and Daw, 2008; Guitart-Masip et al., 2010; Talmi et al., 2008).”
Attention, learning and the value of information (2012, background): “By including an estimate of the uncertainty that arises at each step, one can further calculate the costs of this uncertainty and the beneﬁts of reducing it by obtaining information (Dayan and Daw, 2008; Rothkopf and Ballard, 2010; Sprague and Ballard, 2005; Tatler et al., 2011).”
Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, neutral): “…Thrun, 1992; Thrun & Möller, 1991), cognitive science and neuroscience (Cohen et al., 2007; Daw, O’doherty, Dayan, Seymour, & Dolan, 2006; Dayan & Daw, 2008; K. Friston et al., 2015a; Geana, Wilson, Daw, & Cohen, 2016; Gottlieb, Oudeyer, Lopes, & Baranes, 2013), economic decision-making…”
Cognitive control over learning: creating, clustering, and generalizing task-set structure. (2013, neutral): “…via dopaminergic RL signals (Montague, Dayan, & Sejnowski, 1996), allowing networks to learn to gate responses that are probabilistically most likely to produce a positive outcome and least likely to lead to a negative outcome (Dayan & Daw, 2008; Doya, 2002; Frank, 2005; Houk, 2005; Maia, 2009).”
Perception, Action, and Utility: The Tangled Skein (2012, background): “The brain appears to perform an analogous integration of instantaneous motion energy into an aggregate state estimate one synapse downstream from MT, in the lateral intraparietal area (LIP).”
Error and expectation in language learning: The curious absence of mouses in adult speech (2013, background): “…with similar principles embodied in a wide range of learning models, in which equivalent simulations could be implemented (see e.g. Barlow 2001, Courville et al. 2006, Danks 2003, Dayan & Daw 2008, Gallistel 2003, Kruschke 2008, McLaren & Mackintosh 2000, Pearce & Hall 1980, Sutton & Barto 1998).”
Action understanding as inverse planning. (2009, background): “MDPs are a normative framework for modeling sequential decision making under uncertainty, widely used in rational models of human planning and reinforcement learning (Dayan & Daw, 2008), and in real-world applications in operations research and other ﬁelds (Feinberg & Shwartz, 2002; Puterman, 2005).”
The neuroscience of learning: beyond the Hebbian synapse. (2013, background): “There are several recent, theoretically oriented reviews of the relevant literature (Berridge 2012, Dayan & Daw 2008, Flagel et al. 2011, Schultz 2006, Zhang et al. 2009).”
Revealing Neurocomputational Mechanisms of Reinforcement Learning and Decision-Making With the hBayesDM Package (2016, methodology): “Computational modeling has been particularly useful in the reinforcement-learning and decision-making (RLDM) ﬁelds (Dayan & Daw, 2008; Rangel, Camerer, & Montague, 2008); it has also been integrated into the analysis of neural data, including data from functional magnetic resonance imaging (fMRI)…”
Model-Based and Model-Free Pavlovian Reward Learning: Revaluation, Revision and Revelation (2014, background): “The very different statistical and computational properties (Dayan & Daw, 2008) of model-based versus model-free strategies are a good reason to have both in the same brain.”
Computational models of reinforcement learning: the role of dopamine as a reward signal (2010, background): “These general RL concepts are further explained and elaborated by various authors (Dayan and Daw 2008; Kaelbling et al. 1996; Montague et al. 2004a; Sutton and Barto 1998; Worgotter and Porr 2005; Dayan and Abbott 2001).”
Adolescent risk-taking in the context of exploration and social influence (2021, background): “Evidence suggests that learning to interact with novel environments can be characterized by some form of Bayesian learning (Daw et al., 2005, 2006; Dayan & Daw, 2008; Knill & Pouget, 2004; Marković & Kiebel, 2016; Mathys et al., 2011; Nassar et al., 2010).”
The anatomy of choice: active inference and agency (2013, background): “The question then reduces to how the brain maximizes value (Camerer, 2003; Daw and Doya, 2006; Dayan and Daw, 2008).”
Fast deep reinforcement learning using online adjustments from the past (2018, background): “…memory-like into reinforcement learning algorithms [Blundell et al., 2016a, Pritzel et al., 2017], potentially providing increases in flexibility and learning speed, driven by motivations from the neuroscience literature known as Episodic Control [Dayan and Daw, 2008, Gershman and Daw, 2017].”
Behavioral / Systems / Cognitive Striatal Prediction Error Modulates Cortical Coupling (2010, neutral): “Clearly, there are numerous potential models of such a process (Dayan and Daw, 2008; Dayan and Niv, 2008).”