Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning

By Thomas M. Moerland, Joost Broekens, C. Jonker. 2017

32 citations recorded by the source corpus.

In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In contrast, deep generative models do capture complex high-dimensional outcome distributions. First we discuss why, amongst such models, conditional variational inference (VI) is theoretically most appealing for model-based RL. Subsequently, we compare different VI models on their ability to learn complex stochasticity on simulated functions, as well as on a typical RL gridworld with multimodal dynamics. Results show VI successfully predicts multimodal outcomes, but also robustly ignores these for deterministic parts of the transition dynamics. In summary, we show a robust method to learn multimodal transitions using function approximation, which is a key preliminary for model-based RL in stochastic domains.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, background): “Karl et al. (2016) address mode collapse to a single future by restricting the transition function, (Moerland et al., 2017) focus on multi-modal transitions, and Doerr et al. (2018) stabilize training of purely stochastic models.”
Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes (2017, unknown): “Other approaches use stochastic latent variable inputs to introduce transition stochasticity [12, 30].”
Lipschitz Continuity in Model-based Reinforcement Learning (2018, methodology): “We next applied EM to train a transition model for an RL setting, namely the gridworld domain from Moerland et al. (2017).”
Unsupervised Video Object Segmentation for Deep Reinforcement Learning (2018, background): “More generally, an agent capable of predicting future rewards and observations is extracting useful information that can speed up the learning of a good policy or value function [22].”
Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables (2017, background): “The BNNs with latent variables produce improved probabilistic predictions and these result in better performing policies (Depeweg et al., 2016; Moerland et al., 2017).”
Combating the Compounding-Error Problem with a Multi-step Model (2019, methodology): “(cid:16)(cid:12)(cid:12) (cid:12)(cid:12) (cid:17) We apply this algorithm in a variant of the gridworld (mini-Pacman) domain Moerland et al. [2017].”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, background): “In particular, recent work has extended BNNs with a latent variable model to describe complex stochastic functions [Depeweg et al., 2016, Moerland et al., 2017].”
Efficient exploration with Double Uncertain Value Networks (2017, background): “based on conditional variational inference (Moerland et al., 2017).”
Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction (2017, background): “Modeling this multimodal human response behavior is a core focus of this work; the benefits of using a discrete latent space to learn multimodal distributions has been previously studied in [30], [31], [32].”