Surprising Negative Results for Generative Adversarial Tree Search

By Kamyar Azizzadenesheli, Brandon Yang, Weitang Liu, Emma Brunskill, Zachary Chase Lipton, Anima Anandkumar. 2018

14 citations recorded by the source corpus.

While many recent advances in deep reinforcement learning (RL) rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment model. In this work, we provide an extensive study on the design of deep generative models for RL environments and propose a sample efficient and robust method to learn the model of Atari environments. We deploy this model and propose generative adversarial tree search (GATS) a deep RL algorithm that learns the environment model and implements Monte Carlo tree search (MCTS) on the learned model for planning. While MCTS on the learned model is computationally expensive, similar to AlphaGo, GATS follows depth limited MCTS. GATS employs deep Q network (DQN) and learns a Q-function to assign values to the leaves of the tree in MCTS. We theoretical analyze GATS vis-a-vis the bias-variance trade-off and show GATS is able to mitigate the worst-case error in the Q-estimate. While we were expecting GATS to enjoy a better sample complexity and faster converges to better policies, surprisingly, GATS fails to outperform DQN. We provide a study on which we show why depth limited MCTS fails to perform desirably.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Mastering Atari, Go, chess and shogi by planning with a learned model (2019, neutral): “Surprisingly, and in contrast to previous work [1], even with only 6 simulations per move – fewer than the number of actions – MuZero learned an effective policy and improved rapidly.”
Structured agents for physical construction (2019, methodology): “Here we combine our DQN agent with MCTS, in the spirit of Sutton (1991) and Azizzadenesheli et al. (2018).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “28 However, we found very few papers with this characteristic[17,170,208] — we note that this is not encouraged in the ML community; moreover, negative results reduce the chance of paper acceptance [197].”
The Scientific Method in the Science of Machine Learning (2019, background): “Examples of hypothesis statements in the literature include those in Nguyen et al. (2015); Azizzadenesheli et al. (2018); Recht et al. (2019).”