Learning to predict by the methods of temporal differences

By Richard S. Sutton. 1988

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “That is, after each episode, a single gradient-based update is made to the policy, and the policy optimization does not revisit those trajectory data again.”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “This inaccuracy is further exaggerated by the nature of temporal difference learning (Sutton, 1988), in which an estimate of the value function is updated using the estimate of a subsequent state.”
When to use parametric models in reinforcement learning? (2019, methodology): “Second, we could rely less on bootstrapping by using multi-step returns [Sutton, 1988, van Hasselt and Sutton, 2015, Sutton and Barto, 2018].”
Distributed Prioritized Experience Replay (2018, methodology): “., 2015) with some of the components of Rainbow (Hessel et al., 2017). More speciﬁcally, we used double Q-learning (van Hasselt, 2010; van Hasselt et al., 2016) with multi-step bootstrap targets (cf. Sutton, 1988; Sutton & Barto, 1998; 2017; Mnih et al., 2016) as the learning algorithm, and a du”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, methodology): “Using these values and the predicted rewards, TreeQN then performs a tree backup, mixing the k -step returns along each path in the tree using TD( λ ) (Sutton, 1988; Sutton & Barto, 1998).”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Learning from multi-step boot-strap targets (Sutton 1988; Sutton and Barto 1998), as used in A3C (Mnih et al. 2016), shifts the bias-variance trade-off and helps to propagate newly observed rewards faster to earlier visited states.”
The Predictron: End-To-End Learning and Planning (2016, methodology): “This λ-preturn is analogous to the λ-return in the forward-view TD(λ) algorithm (Sutton, 1988; Sutton & Barto, 1998).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “In light of this, it is tempting to consider how such sequence models can lead to improved performance in RL, which is also concerned with sequential processes (Sutton, 1988).”
Deep Reinforcement Learning with Double Q-Learning (2015, background): “Estimates for the optimal action values can be learned using Q-learning (Watkins, 1989), a form of temporal difference learning (Sutton, 1988).”
Neural Approaches to Conversational AI (2018, neutral): “erence” r Q(s;a;): (2.4) The above update is known as Q-learning (Watkins, 1989), which applies a small change to , controlled by the step-size parameter and computed from the temporal difference (Sutton, 1988). While popular, in practice, Q-learning can be quite unstable and requires many samples b”
Safe and Efficient Off-Policy Reinforcement Learning (2016, background): “Return-based Operators: The λ -return extension (Sutton, 1988) of the Bellman operators considers exponentially weighted sums of n -steps returns: where T π Q − Q is the Bellman residual of Q for policy π .”
Counterfactual Multi-Agent Policy Gradients (2017, methodology): “In this paper, we train critics f (·, θ) on-policy to estimate either Q or V , using a variant of TD(λ) (Sutton 1988) adapted for use with deep neural networks.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “This idea is analogous to bootstrapping in the TD(0) algorithm for reinforcement learning (Sutton, 1988).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ature. The function can generalize this correlation to the beginning of an episode and predict already high reward for states possessing the same feature. Multi-step temporal difference (TD) learning [127, 128] improved both DQNs and policy gradients [47, 80]. AlphaGo and AlphaZero learned to play G”
Meta-Gradient Reinforcement Learning (2018, methodology): “In this work, we are interested in one of the fundamental problems in reinforcement learning: what would be the best form of return for the agent to maximise? Specifically, we propose to learn the return function by treating it as a parametric function with tunable meta-parameters η, for instance in”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “This involves sampling transition tuples (s, a, r, s′) with uniform probability, and applying the temporal difference update (Sutton, 1988; Watkins, 1989):”
Deep Reinforcement Learning framework for Autonomous Driving (2017, methodology): “There are popular algorithms to solve this problem for ﬁnite state-space like Value iteration and Policy iteration [17].”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “The λ-return is the foundation of the entire TD(λ) family of algorithms, which includes the original linear-time algorithm (Sutton, 1988), least-squares formulations (Bradtke & Barto, 1996; Mahmood et al.”