Temporal Difference Learning and TD-Gammon

By G. Tesauro. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, background): “Perhaps the best-known success story of reinforcement learning is TD-gammon, a backgammonplaying program which learnt entirely by reinforcement learning and self-play, and achieved a superhuman level of play [24].”
Investigating Contingency Awareness Using Atari 2600 Games (2012, methodology): “When using function approximation with popular modelfree reinforcement learning algorithms such as SARSA(λ), it is well known (Sutton 1996; Tesauro 1995) that good performance hinges on having access to an appropriate set of basis functions or features.”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Recently, signiﬁcant progress has been made by combining advances in deep learning for learning feature representations (Krizhevsky et al., 2012; Hinton et al., 2012) with reinforcement learning, tracing back to much earlier work of Tesauro (1995) and Bert-sekas & Tsitsiklis (1995).”
One-Shot Imitation Learning (2017, background): “Reinforcement learning has had many successes, including Backgammon [57], helicopter control [39], Atari [35], Go [52], continuous control in simulation [51, 21, 32] and on real robots [40, 30].”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “The use of rich and ﬂexible function approximators such as neural networks in RL dates back many years, e.g. [38, 35, 12, 10].”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Model-free RL has a myriad of applications in games [22, 37], robotics [16, 17], and marketing [18, 38], to name a few.”
PGQ: Combining policy gradient and Q-learning (2016, background): “Reinforcement learning has seen success in several areas including robotics (Lin, 1993; Levine et al., 2015), computer games (Mnih et al., 2013; 2015), online advertising (Pednault et al., 2002), board games (Tesauro, 1995; Silver et al., 2016), and many others.”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Both have been combined with non-linear function approximation (Tesauro, 1995; Williams, 1992), and have achieved remarkable successes on multiple challenging domains; yet, these algorithms still require large amounts of data to determine good policies for any new environment.”
DeepStack: Expert-level artificial intelligence in heads-up no-limit poker (2017, background): “, backgammon (1), checkers (2), chess (3), Jeopardy! (4), Atari video games (5), and go (6).”
Deep Reinforcement Learning with Double Q-Learning (2015, methodology): “This is important because ﬂexi-ble parametric function approximators are often employed in reinforcement learning (see, e.g., Tesauro 1995; Sallans and Hinton 2004; Riedmiller 2005; Mnih et al. 2015).”
Neural Approaches to Conversational AI (2018, positive): “from data. In general, Q(s;a;) may take different parametric forms. For example, in the case of Deep Q-Network (DQN), Q(s;a;) takes the form of deep neural networks, such as multi-layer perceptron (Tesauro, 1995; Mnih et al., 2015), recurrent network (Hausknecht and Stone, 2015; Li et al., 2015), et”
Giraffe: Using Deep Reinforcement Learning to Play Chess (2015, background): “The most famous study in using temporal-diﬀerence reinforcement learning to play zero-sum games is probably Tesauro’s backgammon program TD-Gammon [19] from 1995.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Self-Play (SP) (Tesauro, 1995), the most common centralized training setting involves training a single agent against itself without any extra supervision.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Adversarial games have been proposed in RL with the goal of improving exploration [42, 27], learning goal conditioned policies [36] and generating an automated curriculum [45, 41, 2].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “While there has been signiﬁcant progress in developing deep reinforcement learning algorithms that learn complex skills and scale to high-dimensional observation spaces, such as pixels [1], [2], [3], [4], learning behaviors that generalize to new tasks and objects remains an open problem.”
Deep Exploration via Bootstrapped DQN (2016, background): “Many successful RL applications focus on generalization and planning but address exploration only via ine cient exploration [12] or even none at all [23].”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “The potential power of parametric TD learning was convincingly demonstrated by Tesauro (1992, 1995) when he applied TD(λ) combined with neural networks and self play to obtain ultimately the world’s best backgammon player.”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “One may ask whether it is possible to extend the consistent Bellman operator to Q-value approximation schemes which lack a probabilistic interpretation, such as linear approximation (Sutton 1996), locally weighted regression (Atkeson 1991), neural networks (Tesauro 1995), or even informationtheoreti”