Uncertainty-driven Imagination for Continuous Deep Reinforcement Learning

By Gabriel Kalweit, Joschka Boedecker. 2017

134 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, methodology): “The deterministic model has the same architecture as this ﬁgure but without the inference network.”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “…(Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018), combining the beneﬁts of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018;…”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “There also exists another branch of model-based algorithms where a model is learned from experiences. with which the agent can interact and collect imaginary rollouts (Sutton, 1991), It has also been extended with DRL methods (Weber et al., 2017; Kalweit and Boedecker, 2017).”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “…rapidly degrades with increasing model error (Gu et al., 2016), motivating work that interpolates between different rollout lengths (Buckman et al., 2018), tunes the ratio of real to model-generated data (Kalweit & Boedecker, 2017), or does not rely on model predictions (Heess et al., 2015).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “For example, estimated models (Levine & Koltun, 2013; Gu et al., 2016; Kalweit & Boedecker, 2017) On the control theory side, Dean et al. (2018; 2017) provide strong ﬁnite sample complexity bounds for solving linear quadratic regulator using model-based approach.”
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion (2018, methodology): “Kalweit and Boedecker [17] train a model-free algorithm on a mix of real and imagined data, adjusting the proportion in favor of real data as the Q-function becomes more confident.”
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning (2018, methodology): “We do not use bootstrap to limit model use as MA-BDDPG does.”
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, background): “One possible strategy to generate more diverse experience is to roll out more than a single step from the start state during planning like Gu et al. (2016) and Kalweit and Boedecker (2017).”
Learning Powerful Policies by Using Consistent Dynamics Model (2019, methodology): “Recently, the Model-assisted Bootstrapped DDPG (MA-DDPG, Kalweit & Boedecker, 2017) was proposed to incorporate model-based rollouts into a Deep Deterministic Policy Gradient method.”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “The model-bias problem has motivated work that uses meta-learning (Clavera et al., 2018), interpolation between different horizon predictions (Buckman et al., 2018; Janner et al., 2019), and interpolating between model and real data (Kalweit & Boedecker, 2017).”
γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction (2020, methodology): “…et al., 2018), feeding model-generated data into an otherwise model-free optimizer (Sutton, 1990; Silver et al., 2008; Lampe & Riedmiller, 2014; Kalweit & Boedecker, 2017; Luo et al., 2019), using model predictions to improve the quality of target values for temporal difference learning…”
Mastering Atari with Discrete World Models (2020, background): “, 2018; Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 20”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “(y|x?, θ)]]︸︷︷︸ Total Uncertainty − Ep(θ|D)[H[P (y|x?, θ)]]︸︷︷︸ Expected Data Uncertainty (33) 13TABLE 2: Further information of some UQ methods used in RL. Study Application Goal/Objective UQ method Code Tegho et al. [176] Dialogue management context Dialogue policy optimisation BBB propagation”