Dopamine: generalization and bonuses

By Sham M. Kakade, Peter S. Dayan. 2002

475 citations recorded by the source corpus.

In the temporal difference model of primate dopamine neurons, their phasic activity reports a prediction error for future reward. This model is supported by a wealth of experimental data. However, in certain circumstances, the activity of the dopamine cells seems anomalous under the model, as they respond in particular ways to stimuli that are not obviously related to predictions of reward. In this paper, we address two important sets of anomalies, those having to do with generalization and novelty. Generalization responses are treated as the natural consequence of partial information; novelty responses are treated by the suggestion that dopamine cells multiplex information about reward bonuses, including exploration bonuses and shaping bonuses. We interpret this additional role for dopamine in terms of the mechanistic attentional and psychomotor effects of dopamine, having the computational role of guiding exploration.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Molecular Biology

Citations in the corpus

Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, positive): “riosity-driven learning) grounded into theories of motivation, spontaneous exploration, free play and development in humans [130, 35, 13] as well as in recent ndings in the neuroscience of motivation [104, 52, 83]. As argumented in [96, 12, 8, 62], architectures based on intrinsically motivated lear”
The computational neurobiology of learning and reward. (2006, background): “There is also a theoretical account of why dopamine neurons should respond to novel, neutral stimuli (if they encode a reward prediction error) [46].”
Novelty or Surprise? (2013, methodology): “Models of classical conditioning based on Bayesian methods, including the Kalman ﬁlter, have been proposed that go beyond the account provided by the Rescorla-Wagner model (Dayan et al., 2000; Kakade and Dayan, 2002a; Courville et al., 2004, 2006).”
Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens (2007, background): “, and that these responses have also been modeled using temporal difference algorithm”
Neural mechanisms underlying the induction and relief of perceptual curiosity (2012, background): “In the reinforcement-learning literature, this bias towards the exploration of uncertain or novel options is captured by the concept of an “exploration bonus” that is assigned to uncertain or novel stimuli to promote their exploration (Kakade and Dayan, 2002).”
World Discovery Models (2019, unknown): “…in developmental psychology (White, 1959; Deci & Ryan, 1985; Csikszentmihalyi & Csikszentmihalyi, 1992), neuroscience (Dayan & Balleine, 2002; Kakade & Dayan, 2002; Horvitz, 2000) and machine learning (Oudeyer & Kaplan, 2008; Gottlieb et al., 2013; Schmidhuber, 1991a) that an agent…”
Reinforcement learning in the brain (2009, background): “Kakade and Dayan (2002) addressed this possibility directly, and furthermore suggested that the novelty responses can function as ‘novelty bonuses’ – quantities that are added to other available rewards (rnew t = rt + novelty(St)) and enhance exploration of novel stimuli.”
Computational psychiatry (2011, background): “incorporating such rich priors, even though model-free control can be induced to behave in similar ways by simpler mechanisms [65,66].”
Intrinsic motivations and open-ended development in animals, humans, and robots: an overview (2014, background): “…computational modeling is linking some of these neuroscientiﬁc results to speciﬁc computational mechanisms, e.g., in relation to dopamine (e.g., see the pioneering work of Kakade and Dayan, 2002, and Mirolli et al., 2013) and goal-directed behavior (Baldassare et al., 2013).”
Multiphasic Temporal Dynamics in Responses of Midbrain Dopamine Neurons to Appetitive and Aversive Stimuli (2013, background): “However, the system necessarily has little information about stimulus identity and reward value at short latencies (Mirenowicz and Schultz, 1996; Redgrave et al., 1999; Kakade and Dayan, 2002; Nomoto et al., 2010).”
Reinforcement Learning or Active Inference? (2009, unknown): “, Parkinson’s disease; [50]) and reinforcement learning [51,52].”
Deep Curiosity Search: Intra-Life Exploration Improves Performance on Challenging Deep Reinforcement Learning Problems (2018, methodology): “a high temporal difference error) [28], encouraging the discovery of novel states and creating an instinct to play/explore that may facilitate an animal’s survival by leading to the acquisition of useful skills [26, 29, 30, 31].”
Intrinsically Motivated Reinforcement Learning (2004, background): “Recent studies [2, 3] have focused on the idea that dopamine not only plays a critical role in the extrinsic motivational control of behaviors aimed at harvesting explicit rewards, but also in the intrinsic motivational control of behaviors associated with novelty and exploration.”
Dopamine, Affordance and Active Inference (2012, background): “Dopamine has been implicated in a bewildering variety of processes and pathologies in the human brain; ranging from cortical excitability to attentional deficits [5], [6]; from motor control to akinesia and set switching deficits in Parkinson’s disease [7], [8], [9]; from working memory to schizophr”
From hardware and software to kernels and envelopes: a concept shift for robotics, developmental psychology, and brain sciences (2011, background): “Actually, Kakade and Dayan argued that this interpretation is not incompatible with reward error-signaling hypothesis provided that the model is modified to account for novelty effect (Kakade and Dayan, 2002).”
The Wick in the Candle of Learning (2008, unknown): “In reinforcement learning, a novelty bonus is used to motivate the choice of unexplored strategies (Kakade & Dayan, 2002).”