Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

By Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, Sergey Levine. 2018

351 citations recorded by the source corpus.

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “For example, Chua et al. (2018); Doerr et al. (2018); Hafner et al. (2018); Buesing et al. (2018); Feinberg et al. (2018); Buckman et al. (2018) train models to predict a reward and a distribution over next states, minimizing the negative log-probability of the true next state.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “Variants of this basic algorithm have been proposed in a number of prior works, starting from Dyna (Sutton, 1991) to more recent methods that incorporate deep networks (Heess et al., 2015; Feinberg et al., 2018; Kalweit & Boedecker, 2017; Kurutach et al., 2018).”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “For example, model-based value expansion (MVE [5]) incorporates a fixed depth of predictive models of system dynamics into model-free value function estimation, and later stochastic ensemble value expansion (STEVE [6]) further mitigates the bias due to model mis-specification by dynamically interpol”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “We refer the reader to Feinberg et al. (2018) for further discussion of this approach.”
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion (2018, methodology): “[7] show that the MVE target with rollout horizon H will decrease the target error by a factor of γ .”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “Our approach, contrary to previous methods, is simple: it does not rely on parameter noise exploration, careful reinitialization of the model weights or policy’s entropy, hard to train probabilistic models, and it does not need to address the model distribution mismatch [23, 10, 35].”
Exploring Model-based Planning with Policy Networks (2019, methodology): “Dynamics is also used to obtain better value estimation to speed up training [11, 8, 3].”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “, 2017), MVE (Feinberg et al., 2018), and STEVE (Buckman et al.”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, methodology): “In an alternate line of work, internal models have been used for variance reduction purposes in model-free RL [54, 55], in contrast to our use of MPC.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “These methods require an accurate and reliable model and will typically suffer from modeling bias, hence these models are still limited to short horizon prediction in more complex domains [27, 30, 13, 42].”
Q-Learning: Theory and Applications (2020, methodology): “For instance, Kalweit & Boedecker (2017) describe a method in which a model-free approach of imagined data increasing with the (bootstrap-estimated) uncertainty in the estimated dynamics model [see also Feinberg et al. (2018) and references therein for more examples of hybrid strategies strategies].”
Analogues of mental simulation and imagination in deep learning (2019, neutral): “com Background planning The most popular approach to background planning, Dyna (Figure 4a) [59], uses a model to produce simulated experience in place of real experience [32,53,60].”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “MBVE [74], STEVE [75], and doubly-robust methods [76, 77, 78] use model-based rollouts to obtain more favorable bias-variance trade-offs for off-policy evaluation.”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, methodology): “One approach is to use the model to generate synthetic trajectories and use them in addition to observed transitions while training a policy (Oh et al., 2015; Feinberg et al., 2018).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, positive): “For this reason, some recent approaches have sought to combine the benefits of model-based and model-free methods (Gu et al., 2016; Chebotar et al., 2017; Pong et al., 2018; Feinberg et al., 2018; Englert and Toussaint, 2016).”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, neutral): “MPC is thus limited to short-horizon planning, and this limitation has been noted in prior work (Nagabandi et al., 2018; Feinberg et al., 2018).”
Ready Policy One: World Building Through Active Learning (2020, background): “Other model-based approaches, such as (Kurutach et al., 2018; Feinberg et al., 2018; Hafner et al., 2020), rely on stochastic policies to aid exploration, and inevitably acquire redundant data which reduces sample efficiency.”
Gradients are Not All You Need (2021, methodology): “…model, however, many methods instead learn a value function which determines performance in the future then differentiates though a handful of steps before truncating the trajectory with this learned value [Feinberg et al., 2018, Buckman et al., 2018, Kurutach et al., 2018, Clavera et al., 2020].”