Exploring Model-based Planning with Policy Networks

By Tingwu Wang, Jimmy Ba. 2019

164 citations recorded by the source corpus.

Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in terms of both sample efficiency and asymptotic performance. Despite their initial successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments, being about 3x more sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released in this https URL.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “Our approach to RL is most closely related to prior model-based methods that plan with a learned model (Chua et al., 2018; Wang & Ba, 2020).”
Adaptive Online Planning for Continual Lifelong Learning (2019, background): “proximation is still semantically meaningful for estimating the actual value of the state. V^(s) = 1 log Xn i=1 eV i(s) logn (2) 3.2 Early Planning Termination Past model-based planning procedures [6, 31] run a ﬁxed number of iterations of MPC per timestep before executing an action in the environm”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [67, 17, 54, 51, 59, 70], we choose to build upon Dyna-style approaches that optimize for a policy [64, 66, 73, 32, 26, 28, 44], specifically MBPO [29].”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “ed planning [25, 63], dynamic programming [26, 64], and policy optimization [32, 65, 33, 27, 53]. These advances in MBRL have enabled highly sample efﬁcient learning in widely studied benchmark tasks [30, 29, 66, 67, 28], as well as in a number of challenging robotic control tasks like aggressive dr”
Reinforcement Learning with Augmented Data (2020, methodology): “For baseline methods, we report the best number in POPLIN [42].”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “POPLIN (Wang and Ba, 2019) improves over online planning by self-imitation.”
Model-based Adversarial Meta-Reinforcement Learning (2020, methodology): “Another branch of MBRL produces policies based on model predictive control (MPC), where at each time step the model is used to perform planning over a short horizon to select actions [Chua et al., 2018, Nagabandi et al., 2018, Dong et al., 2019, Wang and Ba, 2019].”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “…datasets has been studied in the context of model-based RL, which learns explicit forward models (Deisenroth & Rasmussen, 2011; Watter et al., 2015; Williams et al., 2015; Chua et al., 2018; Buckman et al., 2018; Wang & Ba, 2019; Kaiser et al., 2019; Hafner et al., 2019; Nagabandi et al., 2020).”
Model-Augmented Actor-Critic: Backpropagating through Paths (2020, methodology): “MPC methods (Nagabandi et al., 2017) have shown to be very effective when the uncertainty of the dynamics is modelled (Chua et al., 2018; Wang & Ba, 2019).”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “…model-free and model-based elements (Nagabandi et al., 2018; Buckman et al., 2018; Pong et al., 2018; Hafez et al., 2019; Sikchi et al., 2022; Wang & Ba, 2020; Clavera et al., 2020; Hansen et al., 2021; Morgan et al., 2021; Bhardwaj et al., 2021; Margolis et al., 2021), many of which are…”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “The most obvious is model-based RL, where it is used to train predictive models that can then be used for planning or policy learning [58, 53, 14, 13, 36, 22, 10, 63, 1].”
Model-Based Offline Planning (2020, background): “BC-trained policies as sampling priors have been looked at by POPLIN [29].”
Mastering Atari with Discrete World Models (2020, methodology): “World models Several model-based agents focus on proprioceptive inputs (Watter et al., 2015; Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018; Wang et al., 2019; Wang and Ba, 2019), model images without using them for planning (Oh et al.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, methodology): “Other contributions focus on the eﬀect of ensembling some or all of the learned models, such as PETS (Chua et al., 2018), STEVE (Buckman et al., 2018), and POPLIN (Wang and Ba, 2019).”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, background): “…in model-based planning (e.g., Chua et al., 2018; Co-Reyes et al., 2018; Finn and Levine, 2017; Silver et al., 2016, 2017, 2018; Sutton, 1991; Wang and Ba, 2020) studies how models can be used to identify good actions either to update the policy or to structure interactions with the…”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Existing works in model-based planning are restricted to short horizons (Chua et al., 2018; Wang & Ba, 2019; Nagabandi et al., 2020).”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, background): “MPC reﬁnes the stochastic model-free policy (the actor) by sampling-based planning [Wang and Ba, 2019, Hong et al., 2019], and the value function (the critic) mitigates the short sight of imaginary model rollouts in MPC planning [Lowrey et al., 2018, Clavera et al., 2020].”