TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play

By G. Tesauro. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

An Actor-Critic Algorithm for Sequence Prediction (2016, methodology): “While RL methods with non-linear function approximators are not new (Tesauro, 1994; Miller et al., 1995), they have recently surged in popularity, giving rise to the ﬁeld of ‘deep RL’ (Mnih et al., 2015).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “While the reinforcement learning algorithms have been an active area of research for decades, the introduction of effective high-capacity function approximators – deep neural networks – into reinforcement learning, along with effective algorithms for training them, has allowed reinforcement learning”
Deep learning in neural networks: An overview (2014, background): “Stone, M. (1974). Cross-validatory choice and assessment o f statistical predictions.”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, background): “This technique was successfully used in prior work to obtain super human performance in a variety of multiplayer games including Backgammon, Go, Chess, Hex, StarCraft 2, Poker [1, 4, 7, 37–39].”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “While the principles of deep reinforcement learning have been known for decades [11, 12], operationalizing them in a practical robotic learning algorithm that can generalize to new objects requires a stable and scalable algorithm and large datasets, as well as careful system design.”
Learning to Act by Predicting the Future (2016, background): “While a sparse scalar reward may be the only feedback available in a board game (Tesauro, 1994; Silver et al., 2016), a multidimensional stream of sensations is a more appropriate model for an organism that is learning to function in an immersive environment (Adolph & Berger, 2006).”
Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings (2021, background): “Most of these successes were achieved in fully observable games such as Backgammon (Tesauro, 1994), Chess (Campbell et al., 2002) and Go (Silver et al., 2016; 2017; 2018).”
Reinforcement Learning: A Survey (1996, neutral): “More recently, Tesauro (1992, 1994, 1995) applied the temporal di(cid:11)erence algorithm to backgammon.”
Infinite-Horizon Policy-Gradient Estimation (2001, background): “This approach has yielded some remarkable empirical successes in a number of different domains, including learning to play checkers (Samuel, 1959), backgammon ( Tesauro, 1992, 1994 ), and chess (Baxter, Tridgell, & Weaver, 2000), job-shop scheduling (Zhang & Dietterich, 1995) and dynamic channel all”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “3 Gradient Descent-Based NNs for RL Perhaps the most well-known RL application is Tesauro’s backgammon player [280] from 1994 which learned to achieve the level of human world champions, by playing against itself.”
Learning Dexterous Manipulation Policies from Experience and Imitation (2016, background): “The idea of learning policies without having models still dominates RL, and forms the basis of the most remarkable success stories, both old (Tesauro 1994) and new (Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland and Ostrovski 2015).”
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (2017, methodology): “Our approach is something of a throwback to the approach used by Tesauro (1994) to develop TD-Gammon more than two decades ago.”
"Other-Play" for Zero-Shot Coordination (2020, background): “A popular way of constructing strategies for MG with unknown opponents is self-play (or “self-training”), (Tesauro, 1994).”
Decision-Theoretic Planning: Structural Assumptions and Computational Leverage (1999, background): “…of neural-network representations of value functions, as frequently adopted within thereinforcement learning community (Bertsekas & Tsitsiklis, 1996; Tesauro, 1994),54 nor havewe discussed the partitioning of continuous state spaces often addressed in reinforcementlearning (Moore & Atkeson, 1995).”
Improving Action Selection in MDP's via Knowledge Transfer (2005, background): “While largestate sets have seen much work in recent research (Tesauro 1994; Crites & Barto 1996; Stone & Sutton 2001), largeactionsets have been explored to but a limited extent (Santamaria, Sutton, & Ram 1997; Gaskett, Wettergreen, & Zelinsky 1999).”