Learning to Play Chess Using Temporal Differences
By Jonathan Baxter, Andrew Tridgell, Lex Weaver. 2000
136 citations recorded by the source corpus.
In this paper we present TDLEAF(λ), a variation on the TD(λ) algorithm that enables it to be used in conjunction with game-tree search. We present some experiments in which our chess program “KnightCap” used TDLEAF(λ) to learn its evaluation function while playing on Internet chess servers. The main success we report is that KnightCap improved from a 1650 rating to a 2150 rating in just 308 games and 3 days of play. As a reference, a rating of 1650 corresponds to about level B human play (on a scale from E (1000) to A (1800)), while 2150 is human master level. We discuss some of the reasons for this success, principle among them being the use of on-line, rather than self-play. We also investigate whether TDLEAF(λ) can yield better results in the domain of backgammon, where TD(λ) has previously yielded striking success.
Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yuxi Li
- Yoshua Bengio
- Timothy P. Lillicrap
- David Silver
- Chelsea Finn
- Jimmy Ba
In Economics and Econometrics
- Matthew Lai
- David Silver
- Timothy P. Lillicrap
- Arthur Guez
- Tomoyuki Kaneko
- Aja Huang
- Marc Lanctot
- Thore Graepel
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Yuxi Li
- Timothy P. Lillicrap
- David Silver
- Jimmy Ba
- Shimon Whiteson
- R´emi Munos
Citations in the corpus
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (2017, background): “KnightCap (6) evaluated positions by a neural network that used an attack-table based on knowledge of which squares are attacked or defended by which pieces.”
- Temporal-difference search in computer Go (2012, unknown): “The heuristic function is a linear combination of binary features, and can be learnt offline by TD learning and self-play (Baxter et al. 2000; Schaeffer et al. 2001; Buro 1999).”
- Large-Scale Optimization for Evaluation Functions with Minimax Search (2014, methodology): “However, strong human players have ratings of more than 3, 000 points at ICC, and this difference means these programs have not reached the top level of chess programs; that is, evaluation functions tuned by reinforcement learning have not yet reached the level of the best-handcrafted evaluation fun”
- Mastering the game of Go without human knowledge (2017, methodology): “Selfplay reinforcement learning approaches have achieved high levels of perfor mance in other games: chess 49–51 , checkers 52 , backgammon 53 , othello 54 , Scrabble 55 and most recently poker 56 .”
- DeepChess: End-to-End Deep Neural Network for Automatic Learning in Chess (2016, unknown): “Although reinforcement learning has also been applied to chess [1,10], the resulting programs exhibit a playing strength at a human master level at best, which is substantially lower than the grandmaster-level state-of-the-art chess programs.”
- Genetic algorithms for mentor-assisted evaluation function optimization (2008, unknown): “Although temporal difference learning has also been applied to chess [4], the results showed that after three days of learning, the playing strength of the program was merely 2150 Elo (see Appendix B for a description of the Elo rating system), which is a very low rating for a chess program.”