Time-Derivative Models of Pavlovian Reinforcement

By Richard S. Sutton, Andrew Barto. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Parametric TD methods have also been developed as models of animal learning (e.g., Sutton & Barto 1990, Klopf 1988, Ludvig, Sutton & Kehoe 2012) and as models of the brain’s reward systems (Schultz, Dayan & Montague 1997), where they have been particularly influential (e.g., Niv & Schoenbaum 2008,…”
Modeling functions of striatal dopamine modulation in learning and planning. (2001, methodology): “Such eligibility traces were introduced to explain how animals learn to associate sensory events that are separated by a delay period (Sutton and Barto, 1990).”
Evolving neuromodulatory topologies for reinforcement learning-like problems (2007, methodology): “Following, an overview of the algorithm is given.”
Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective (2008, background): “The impact of RL was initially felt in research on classical and instrumental conditioning (Barto & Sutton, 1981; Sutton & Barto, 1990; Wickens, Kotter, & Houk, 1995).”
The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. (2002, methodology): “This algorithm has theoretical roots in both artificial intelligence (Sutton & Barto, 1998) and animal learning theory (Sutton & Barto, 1990) and is a generalization of the Rescorla–Wagner learning rule (Rescorla & Wagner, 1972) to the continuous time domain.”
Reinforcement Learning and Causal Models (2017, background): “…is positive when cue B is paired with cue A (since cue A has a positive value), and this error signal drives learning of a positive value for cue B (Sutton & Barto, 1990).2 Despite these successes, the TD model is still essentially an implementation of Thorndike’s law of e↵ect, and hence fails to…”
Evaluating the TD model of classical conditioning (2012, methodology): “Although an obvious simplification, this approach, in combination with an appropriate learning rule, can accomplish a surprisingly wide range of real-time learning phenomena (Sutton & Barto, 1981, 1990).”
Synaptic Plasticity Forms and Functions. (2020, unknown): “…allows many models to accurately capture details of classical conditioning experiments, including the linking of seconds-long CS-US intervals and the production of the useful predictive responses, even with short ETs (Sutton 1988, Sutton & Barto 1990, Montague et al. 1996, Ludvig et al. 2008).”
TD models of reward predictive responses in dopamine neurons (2002, methodology): “Similar hypothetical temporal stimulus representations have also been referred to as " complete serial compound stimulus " (Sutton & Barto, 1990) or " spectral”
Getting formal with dopamine and reward. (2002, unknown): “…that prediction errors, rather than simple stimulus-reinforcer pairings, play a crucial role in learning situations such as animal conditioning (Kamin, 1969), human conditioning (Martin and Levey, 1991), causal learning (Dickinson, 2001), and artificial network learning (Sutton and Barto, 1990).”
Dopamine reward prediction errors reflect hidden state inference across time (2017, unknown): “TD learning uses RPEs to update the weights of task-related features, which were classically represented as a cascade of sub-states (the “complete serial compound” or CSC) that track elapsed time following stimulus onset 1,5 .”
Reinforcement learning in the brain (2009, background): “…Pavlovian conditioning (Yerkes & Morgulis, 1909) is considered as a prototypical instance of prediction learning — learning the predictive relationships between events in the environment such as the fact that the scent of home-cooking usually predicts a tasty meal (e.g. Sutton and Barto (1990)).”
Reinforcement Learning for Learning Rate Control (2017, methodology): “The actor network will be updated using the estimated goodness of a t , and the critic network will be updated by minimizing temporal difference (TD) [ Sutton and Barto, 1990 ] error.”
Multi-timescale nexting in a reinforcement learning robot (2011, methodology): “However, TD( λ ) has also been used as a model of classical conditioning, where the predictions are shorter term and where more than one signal might be viewed as a reward (Sutton & Barto, 1990).”
Computational Models of Neuromodulation (1998, methodology): “…as learning develops, compatible with experimental data (Schultz et al., 1993), and following the general idea of temporal difference learning (Sutton & Barto, 1990): Interestingly, this model chooses to label weights explicitly with space ( i , origin of cortical activity) and time ( ¿ ,…”