Model-Based Planning in Discrete Action Spaces

By Mikael Henaff, William F. Whitney, Yann LeCun. 2017

17 citations recorded by the source corpus.

Planning actions using learned and differentiable forward models of the world is a general approach which has a number of desirable properties, including improved sample complexity over model-free RL methods, reuse of learned models across different tasks, and the ability to perform efficient gradient-based optimization in continuous action spaces. However, this approach does not apply straightforwardly when the action space is discrete, which may have limited its adoption. In this work, we introduce two discrete planning tasks inspired by existing question-answering datasets and show that it is in fact possible to effectively perform planning via backprop in discrete action spaces using two simple yet principled modifications. Our experiments show that this approach can significantly outperform model-free RL based methods and supervised imitation learners.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Learning and Querying Fast Generative Models for Reinforcement Learning (2018, methodology): “Therefore, we take a heurisitic approach similar to Henaff et al. (2017) (and related to Bengio et al., 2013): Instead of feeding the sampled one-hot environment action to the model, we can instead directly feed the probability vector π r ( a t (cid:48) | s t (cid:48) | t ) into the environment…”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, methodology): “[48] learn models of discrete actions environments, and exploit the effective differentiability of the model with respect to the actions by applying continuous control planning algorithms to derive a plan; Schmidhuber [49] uses an environment model to turn environment cost minimization into a networ”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “DistGBP (Henaff et al., 2017; 2019) uses model gradients for online planning in simple tasks.”
Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018, background): “A variety of model-based work in robotic and non-robotic domains has used reward estimation [14, 15, 16, 17, 18, 19].”
Universal Planning Networks (2018, methodology): “Henaff et al. (2017) adopt gradient-based trajectory optimization for model-based planning in discrete action spaces, but rely on representations learned from unsupervised pretraining.”