Deep Variational Reinforcement Learning for POMDPs

By Maximilian Igl, Luisa M. Zintgraf, Tuan Anh Le, Frank D. Wood, Shimon Whiteson. 2018

308 citations recorded by the source corpus.

Many real-world sequential decision making problems are partially observable by nature, and the environment model is typically unknown. Consequently, there is great need for reinforcement learning methods that can tackle such problems given only a stream of incomplete and noisy observations. In this paper, we propose deep variational reinforcement learning (DVRL), which introduces an inductive bias that allows an agent to learn a generative model of the environment and perform inference in that model to effectively aggregate the available information. We develop an n-step approximation to the evidence lower bound (ELBO), allowing the model to be trained jointly with the policy. This ensures that the latent state representation is suitable for the control task. In experiments on Mountain Hike and flickering Atari we show that our method outperforms previous approaches relying on recurrent neural networks to encode the past.

Subfields: Artificial Intelligence, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, background): “…2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018), and pure video prediction without planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016;…”
Causally Correct Partial Models for Reinforcement Learning (2020, neutral): “…next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019; Janner et al., 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “The idea of learning general world models to support decision-making is probably one of the most pervasive ideas in AI research, [30, 11, 41, 42, 14, 43, 44, 45, 46, 33, 30, 47, 48, 3, 49, 50, 51, 52].”
Learning to Drive in a Day (2018, methodology): “Other research has looked at better transformation of observations into states, typically using an RNN [39], [40], as well as methods to perform multi-step planning, as in [41].”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “We use a variational approach related to Igl et al. (2018) to estimate belief over the task.”
POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, background): “Within RL, recent decision-aware optimization efforts have explored partially-observed problems in both model-free (Karkus et al., 2017) and model-based settings (Igl et al., 2018).”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, methodology): “…been used to play video games [Oh et al., 2015, Chiappa et al., 2017]; Ha and Schmidhuber [2018] trained agents in a “dreamland” built using RNNs; Igl et al. [2018] utilized RNNs to characterize belief states in situations with partially observable dynamics; and Neitz et al. [2018] trained a…”
The Value Equivalence Principle for Model-Based Reinforcement Learning (2020, background): “This is an instantiation of the problem known as representation learning [43, 21, 9, 18, 45, 44, 19, 7].”
Mastering Atari with Discrete World Models (2020, background): “, 2018; Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 20”
Context-Specific Representation Abstraction for Deep Option Learning (2021, methodology): “More explicit methods for modeling belief states have been considered, for example, as outlined in Igl et al. (2018).”
Neural belief states for partially observed domains (2018, result): “Similar observations were made in [6], where the belief state is represented by a collection of particles.”
Flexible Diffusion Modeling of Long Videos (2022, background): “In near term future work we will add action (and reward) explicitly to the architecture and training objective transforming our video generative model into a vision-based world model in the reinforcement learning sense [16, 18].”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “…2020] or learning state representations for improving the sample sample efﬁciency of RL algorithms using reconstruction-based [Corneil et al., 2018, Igl et al., 2018, Kurutach et al., 2018, Gregor et al., 2019, Gelada et al., 2019] or non-reconstruction-based methods [Laskin et al., 2020, Zhang…”
Block Contextual MDPs for Continual Learning (2021, background): “Our work is also related to the general problem of training a policy on Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998; Igl et al., 2018; Zhang et al., 2019; Han et al., 2019; Hafner et al., 2019) that capture both nonstationarity and rich observation settings.”
Learning Causal State Representations of Partially Observable Environments (2019, methodology): “[21, 38] use deep variational methods to learn a probability distribution over states, i.e. belief states, and use the belief states for policy optimization with actor-critic [21] and planning [38].”
Analogues of mental simulation and imagination in deep learning (2019, unknown): “[24,25]), and to transform high-dimensional perceptual observations to a more useful representational format.”
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, methodology): “Previous works have typically observed that mixed deterministic-stochastic state space models are more effective [16, 20, 4] than fully stochastic ones.”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, methodology): “Several deep learning approaches to model-free reinforcement learning (Igl et al., 2019) and model learning for planning (Tschiatschek et al., 2018) in partially observable Markov decision processes have recently been proposed and utilise approximate variational inference methods.”