The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach

By Iulian Serban, Chinnadhurai Sankar, Michael Pieper, Joelle Pineau, Yoshua Bengio. 2018

30 citations recorded by the source corpus.

Deep reinforcement learning has recently shown many impressive successes. However, one major obstacle towards applying such methods to real-world problems is their lack of data-efficiency. To this end, we propose the Bottleneck Simulator: a model-based reinforcement learning method which combines a learned, factorized transition model of the environment with rollout simulations to learn an effective policy from few examples. The learned transition model employs an abstract, discrete (bottleneck) state, which increases sample efficiency by reducing the number of model parameters and by exploiting structural properties of the environment. We provide a mathematical analysis of the Bottleneck Simulator in terms of fixed points of the learned policy, which reveals how performance is affected by four distinct sources of error: an error related to the abstract space structure, an error related to the transition model estimation variance, an error related to the transition model estimation bias, and an error related to the transition model class bias. Finally, we evaluate the Bottleneck Simulator on two natural language processing tasks: a text adventure game and a real-world, complex dialogue response selection task. On both tasks, the Bottleneck Simulator yields excellent performance beating competing approaches.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Adversarial Deep Reinforcement Learning in Portfolio Management (2018, methodology): “A different approach for data preprocessing is to combine HMM with reinforcement learning, which is to extract the states beneath the fluctuated prices and learning directly from them [27].”
Value Propagation Networks (2018, methodology): “Model-based planning with end-to-end architectures has recently shown promising results on a variety of tasks and environments, often using Deep Reinforcement Learning as the algorithmic framework (Silver et al., 2016; Oh et al., 2017; Weber et al., 2017; Groshev et al., 2017; Farquhar et al., 2017;”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, background): “[56, 26, 20, 67, 89, 14], these extensions have not been investigated widely in the multi-agent domain.”