Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

By Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee. 2018

370 citations recorded by the source corpus.

Integrating model-free and model-based approaches in reinforcement learning has the potential to achieve the high performance of model-free algorithms with low sample complexity. However, this is difficult because an imperfect dynamics model can degrade the performance of the learning algorithm, and in sufficiently complex environments, the dynamics model will almost always be imperfect. As a result, a key challenge is to combine model-based approaches with model-free learning in such a way that errors in the model do not degrade performance. We propose stochastic ensemble value expansion (STEVE), a novel model-based technique that addresses this issue. By dynamically interpolating between model rollouts of various horizon lengths for each individual example, STEVE ensures that the model is only utilized when doing so does not introduce significant errors. Our approach outperforms model-free baselines on challenging continuous control benchmarks with an order-of-magnitude increase in sample efficiency, and in contrast to previous model-based approaches, performance does not degrade in complex environments.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “(2018); Buesing et al. (2018); Feinberg et al.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “…the beneﬁts of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018; Wayne et al., 2018; Igl et al., 2018; Srinivas et al., 2018), and pure video prediction without…”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “…while these prior methods typically require additional machinery to work well, such as ensembles in the online setting (Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism mechanisms in the ofﬂine setting (Yu et al., 2020; Kidambi et al., 2020; Argenson &…”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “For example, model-based value expansion (MVE [5]) incorporates a fixed depth of predictive models of system dynamics into model-free value function estimation, and later stochastic ensemble value expansion (STEVE [6]) further mitigates the bias due to model mis-specification by dynamically interpol”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “, 2018) or valuebased methods (Buckman et al., 2018).”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “While there is no natural way to give a policy access to the ground truth model, we allow the policies to train on future trajectories generated via the ground truth model, in similar fashion to algorithms that learn a model for this purpose [5, 18], in order to help facilitate fair comparisons to m”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “For model-based methods, we compare to PETS (Chua et al., 2018), which does not perform explicit policy learning, but directly uses the model for planning; STEVE (Buckman et al., 2018), which also uses short-horizon model-based rollouts, but incorporates data from these rollouts into value estimatio”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, background): “Buckman et al. (2018) reported the reduction in overestimation originating from ensembling in model-based learning.”
Challenges of Real-World Reinforcement Learning (2019, methodology): “A common approach is to learn ensembles of transition models and use various sampling strategies from those models to drive exploration and improve sample efﬁciency (Hester & Stone, 2013; Chua et al., 2018; Buckman et al., 2018).”
Exploring Model-based Planning with Policy Networks (2019, methodology): “Dynamics is also used to obtain better value estimation to speed up training (Gu et al., 2016; Feinberg et al., 2018; Buckman et al., 2018).”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “, 2018), and STEVE (Buckman et al., 2018) learn dynamics for multi-step Q-learning from a replay buffer.”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, methodology): “In an alternate line of work, internal models have been used for variance reduction purposes in model-free RL [54, 55], in contrast to our use of MPC.”
Deep reinforcement learning: a survey (2020, background): “Stochastic ensemble value expansion (Buckman et al., 2018) ensures that the model is used only when doing so does not introduce significant errors by dynamical interpolation between model rollouts of various horizon lengths.”
The Measure of Intelligence (2019, neutral): “[10]), in particular in the context of RL for games such as Atari games or Minecraft (e.”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “For example, Clavera et al. (2020) and Buckman et al. (2018); Lowrey et al. (2019) use a learned model to improve policy and value learning, respectively, through generated trajectories.”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, background): “These settings include, but are not limited to, regression, language modeling (Gal, 2016), image segmentation (Gustafsson et al., 2019), active learning (Settles, 2012) and reinforcement learning (Buckman et al., 2018; Chua et al., 2018).”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “To address some of the critical issues in model-based learning (Langlois et al., 2019), recent methods such as MBPO combine a model ensemble with a carefully controlled rollout horizon to obtain better performance (Janner et al., 2019; Buckman et al., 2018).”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “Figure 2 shows our results in comparison to MBPO and STEVE, which evaluate on the MuJoCo tasks in the OpenAI gym (Brockman et al., 2016) that are mostly the standard v2 tasks with early termination and alive bonuses, and with a truncated observation space for the humanoid and ant that discards the i”