Model-Based Reinforcement Learning via Meta-Policy Optimization

By Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, Pieter Abbeel. 2018

259 citations recorded by the source corpus.

Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “e to use priors that make the planner stay close to policy outputs, which is problematic in changing worlds, when the policy is not accurate. [2, 24] learn dynamics models and perform MPC using them. [7, 15, 18] utilize model ensembles to reduce model overﬁtting for policy learning. Integrating AOP ”
MOPO: Model-based Offline Policy Optimization (2020, background): “Uncertainty quantiﬁcation, a key ingredient to our approach, is critical to good performance in model-based RL both theoretically [62, 73, 43] and empirically [11, 6, 49, 38, 7], in optimal control [61, 2, 33].”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “Ensembles have shown to be effective in preventing a policy or planning procedure from exploiting such inaccuracies (Rajeswaran et al., 2017; Kurutach et al., 2018; Clavera et al., 2018; Chua et al., 2018).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Clavera et al. (Clavera et al., 2018) extended this method with meta-policy optimization and improve the robustness to model error.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, neutral): “We refer to [Kurutach et al., 2018, Clavera et al., 2018, Sun et al., 2018b, Chua et al., 2018, Sanchez-Gonzalez et al., 2018, Pascanu et al., 2017, Khansari-Zadeh and Billard, 2011, Luo et al., 2018] and the reference therein for recent work on model-based RL. Off-policy reinforcement learning…”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…bias, such as the use of multiple dynamics models as an ensemble (Chua et al., 2018; Kurutach et al., 2018; Janner et al., 2019), meta-learning (Clavera et al., 2018), energy-based regularizer (Boney et al., 2019), game-theoretic framework (Rajeswaran et al., 2020), and explicit penalty for…”
ProMP: Proximal Meta-Policy Search (2018, background): “, 2018a), model-based RL (Clavera et al., 2018; Saemundsson et al., 2018), learning in competitive environments (Al-Shedivat et al.”
Skill-based Meta-Reinforcement Learning (2022, unknown): “…2016; Wang et al., 2017; Finn et al., 2017; Yu et al., 2018; Rothfuss et al., 2019; Gupta et al., 2018; Vuorio et al., 2018; Nagabandi et al., 2019; Clavera et al., 2019; 2018; Rakelly et al., 2019; Vuorio et al., 2019; Yang et al., 2019; Zintgraf et al., 2019; Humplik et al., 2019; Zintgraf et…”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “Tools like ensembles, probabilistic models, planning over shorter horizons, and meta-learning have been used to achieved such performance (Kurutach et al., 2018; Chua et al., 2018; Clavera et al., 2018).”
Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, methodology): “Gradient-based approaches to meta-RL seek policy parameters that can be updated to the current task with a few gradient steps [9, 12, 29, 4].”
Offline Meta Reinforcement Learning (2020, background): “Gradient based approaches to meta-RL seek policy parameters that can be updated to the current task with a few gradient steps [12, 16, 31, 4], allowing for quick learning.”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, background): “Much of the focus in prior work has been on improving the efficiency and generalization of RL algorithms to make real-world training feasible, or else on utilizing simulation and transferring policies into the real world (Tzeng et al., 2015; Tobin et al., 2017; Peng et al., 2017; Clavera et al., 201”
Dynamics-aware Embeddings (2019, background): “Another line of work couples the process of learning a model of the environment with training a policy on imagined rollouts (Sutton, 1991; Deisenroth & Rasmussen, 2011; Ha & Schmidhuber, 2018; Clavera et al., 2018; Kaiser et al., 2019; Henaff et al., 2019).”
Learning scheduling algorithms for data processing clusters (2018, methodology): “One viable approach might be to use meta learning [22, 27, 29], which”
Model-augmented Prioritized Experience Replay (2022, methodology): “Thus far, various MbRL methods (Kurutach et al., 2018; Luo et al., 2018; Clavera et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020; Clavera et al., 2020; Silver et al., 2017; Schrittwieser et al., 2020; Hafner et al., 2020a; Kaiser et al., 2019) have been proposed, but the common strategy…”
Ready Policy One: World Building Through Active Learning (2020, background): “…progress in MBRL, with success in Atari games (Kaiser et al., 2020; Schrittwieser et al., 2019), and dexterous manipulation (Nagabandi et al., 2019), while progress has been made on continuous control benchmarks (Kurutach et al., 2018; Chua et al., 2018; Clavera et al., 2018; Janner et al., 2019).”
Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy (2022, methodology): “…methods based on MAML (Finn, Abbeel, and Levine 2017) usually learn a pre-trained model from a set of training environments and adapt the pre-trained model to a new environment via several policy gradient steps (Rothfuss et al. 2019; Stadie et al. 2018; Nagabandi et al. 2019; Clavera et al. 2018).”
Neural-Fly enables rapid learning for agile flight in strong winds (2022, methodology): “In the first class, the adaptation phase adapts the whole neural network model, typically using one or more gradient descent steps (30, 32, 34, 36).”