Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning

By Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L. Lewis, Xiaoshi Wang. 2014

373 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Trust Region Policy Optimization (2015, methodology): “The results of the vine and single path algorithms are summarized in Table 1, which also includes an expert human performance and two recent methods: deep Q-learning (Mnih et al., 2013), and a combination of Monte-Carlo Tree Search with supervised training (Guo et al., 2014), called UCC-I.”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “In recent years, deep reinforcement learning has achieved many impressive results, including playing Atari games from raw pixels (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015), and acquiring advanced manipulation and locomotion skills (Levine et al., 2016; Lillicrap et al., 2015;…”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Notable examples are training agents to play Atari games based on raw pixels (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015a) and to acquire advanced manipulation skills using raw sensory inputs (Levine et al., 2015; Lillicrap et al., 2015; Watter et al., 2015).”
End-to-End Training of Deep Visuomotor Policies (2015, methodology): “CNNs have also been trained to play video games with temporal difference learning and Monte Carlo tree search [9, 26].”
Thinking Fast and Slow with Deep Learning and Tree Search (2017, methodology): “[11], the learning target used was simply the move chosen by MCTS.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “While GATS is an appealing and ﬂexible RL paradigm, it suffers from high computation cost due to modeling in image space and due to MCTS. Potentially, we could mitigate this overhead with parallelization or distilled policy methods (Guo et al., 2014) by approximating with a smaller network.”
Value Iteration Networks (2016, methodology): “In future work we intend to learn different planning computations, based on simulation [11], or optimal linear control [27], and combine them with reactive policies, to potentially develop RL solutions for task and motion planning [12].”
Prioritized Experience Replay (2015, methodology): “…have been several proposed methods for playing Atari with deep reinforcement learning, including deep Q-networks (DQN) (Mnih et al., 2013; 2015; Guo et al., 2014; Stadie et al., 2015; Nair et al., 2015; Bellemare et al., 2016), and the Double DQN algorithm (van Hasselt et al., 2016), which is…”
Dueling Network Architectures for Deep Reinforcement Learning (2015, background): “The results of Schaul et al. (2016) are the current published state-of-the-art.”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, methodology): “[11] used the ALE-emulator for making action-conditional predictions with slow UCT [15], a Monte-Carlo tree search method, to generate training data for a fast-acting CNN, which outperformed DQN on several domains.”
Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (2015, methodology): “Guo et al. (2014) use DAGGER with expert guidance provided by Monte-Carlo Tree Search (MCTS) policies to train a deep neural network that improves on the original DQN’s performance.”
Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models (2015, methodology): “Methods based on Monte-Carlo search can also scale gracefully to complex domains [20], and indeed previous work has applied such techniques to the task of playing Atari games from screen images [21].”
Deep learning in neural networks: An overview (2014, methodology): “Even better results are achieved by using (slow) Monte Carlo tree planning to train comparatively fast deep NNs (Guo et al., 2014).”
Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
Learning Multiagent Communication with Backpropagation (2016, background): “3 Rel ted W rk Our model co bi es a deep network with reinf c ment learning [8, 20, 13].”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “…(Levine et al., 2015; Assael et al., 2015; Wat-ter et al., 2015), visual attention (Ba et al., 2015), and the Atari learning environment (ALE) (Guo et al., 2014; Mnih et al., 2015; Stadie et al., 2015; Wang et al., 2015; Schaul et al., 2016; van Hasselt et al., 2016; Oh et al., 2015;…”