Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

By Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, E. Todorov, Igor Mordatch. 2018

268 citations recorded by the source corpus.

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex simulated control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, background): “For instance, the model may predict a shortcut that does not actually exist; using this to then steer behaviour results in experience that is both suitable to correct the error in the model, and that yields the kind of directed, temporally consistent behaviour typically sought for exploration purpos”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “In our work, we consider online learning in the same style as [20], where both acting and learning must occur on a per-timestep basis, and there are no episodes that reset the state.”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “Unknown state-action detector (USAD): In order to partition the state-action space into known and unknown regions, we use uncertainty quantification [69, 70, 71, 72].”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “By using an uncertainty estimation algorithm like bootstrap ensembles [43, 4, 37], we can estimate u ( s , a ) .”
Online Meta-Learning (2019, neutral): “Prior works have also considered settings that combine joint training across tasks with task-speciﬁc adaptation (Barto et al., 1995; Lowrey et al., 2019), but have not explicitly employed meta-learning.”
Deep Dynamics Models for Learning Dexterous Manipulation (2019, methodology): “As derived by recent model-predictive path integral work [28, 16], this general update rule takes the following form for time step t, reward-weighting factor γ, and reward Rk from each of the N predicted trajectories:”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “Also assuming access to the dynamics, POLO (Lowrey et al., 2018) plans to explore by learning a value ensemble.”
Model-Based Offline Planning (2020, background): “MBOP and other model-based approaches [11, 12, 13, 14, 15, 5] are very different.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Lu et al. (2019) and Lowrey et al. (2018) highlight the beneﬁts of both planning and model-free RL for lifelong agents but require highly accurate world models.”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, background): “MPC reﬁnes the stochastic model-free policy (the actor) by sampling-based planning [Wang and Ba, 2019, Hong et al., 2019], and the value function (the critic) mitigates the short sight of imaginary model rollouts in MPC planning [Lowrey et al., 2018, Clavera et al., 2020].”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “Finally, while learning a terminal value function for MPC has previously been proposed (Negenborn et al., 2005; Lowrey et al., 2019; Bhardwaj et al., 2020; Hatch & Boots, 2021), we are (to the best of our knowledge) the ﬁrst to jointly learn model and value function through TD-learning in…”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “, 2018) or path integral optimal control (Williams et al., 2015; Lowrey et al., 2019), as well as more sophisticated models and planning or control methods would likely improve overall performance further, but is outside the scope of this work.”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “…and proposed additional components to enhance efficiency and stability, such as the use of model predictive control (Williams et al., 2017; Lowrey et al., 2019; Nagabandi et al., 2019), uncertainty quantification through Bayesian models (Deisenroth & Rasmussen, 2011), and ensembles of…”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “Finally, since our approach returns a stochastic policy, dynamics model, and value function could use model-predictive control (MPC) for better performance at test time, similar to (Lowrey et al., 2018; Hong et al., 2019).”
Planning to Explore via Self-Supervised World Models (2020, methodology): “Osband et al. (2018); Lowrey et al. (2018) use ensembles of Q-functions to track the posterior of the value functions.”