Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games

By Xiaoxiao Guo, Satinder Singh, Richard L. Lewis, Honglak Lee. 2016

56 citations recorded by the source corpus.

Monte Carlo Tree Search (MCTS) methods have proven powerful in planning for sequential decision-making problems such as Go and video games, but their performance can be poor when the planning depth and sampling trajectories are limited or when the rewards are sparse. We present an adaptation of PGRD (policy-gradient for reward-design) for learning a reward-bonus function to improve UCT (a MCTS algorithm). Unlike previous applications of PGRD in which the space of reward-bonus functions was limited to linear functions of hand-coded state-action-features, we use PGRD with a multi-layer convolutional neural network to automatically learn features from raw perception as well as to adapt the non-linear reward-bonus function parameters. We also adopt a variance-reducing gradient method to improve PGRD's performance. The new method improves UCT's performance on multiple ATARI games compared to UCT without the reward bonus. Combining PGRD and Deep Learning in this way should make adapting rewards for MCTS algorithms far more widely and practically applicable than before.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

Value Prediction Network (2017, background): “Another line of work [23, 4, 8, 30] uses observation-prediction models not for planning, but for improving exploration.”
Value Iteration Networks (2016, neutral): “In addition, the effects of modelling errors in VINs can be mitigated by training the network end-to-end, similarly to the methods in [13, 11].”
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, methodology): “It works for Q-learning [20], for policy gradient algorithms like A3C [21] and for model-based RL [14].”
Probabilistically safe policy transfer (2017, methodology): “For example, reinforcement learning methods have been used to train a computer to outperform human performance in 49 Atari games [14], [18].”
On Learning Intrinsic Rewards for Policy Gradient Methods (2018, methodology): “Guo et al. [2016] extended the scalability of PGRD to high-dimensional image inputs in Atari 2600 games and used the intrinsic reward as a reward bonus to improve the performance of the Monte Carlo Tree Search algorithm using the Atari emulator as a model for the planning.”
What Can Learned Intrinsic Rewards Capture? (2019, methodology): “…return ( G life ) given any randomly initialised agents and possibly non-stationary tasks drawn from some distribution p ( T ) . been follow-up works (Sorg et al., 2010; Guo et al., 2016) that use a gradient-based method, they consider a non-parameteric policy using Monte-Carlo Tree Search (MCTS).”
Automatic Goal Generation for Reinforcement Learning Agents (2017, background): “, 2016), to outperform humans in 49 Atari games (Guo et al., 2016; Mnih et al., 2015), and to perform a variety of difficult robotics tasks (Lillicrap et al.”
LIIR: Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning (2019, background): “The optimal intrinsic reward problem has been introduced in [13] for single agent learning scenarios and studied in some recent RL approaches [14, 15, 16].”