Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning

By Yevgen Chebotar, Karol Hausman, Marvin Zhang, G. Sukhatme, Stefan Schaal, Sergey Levine. 2017

168 citations recorded by the source corpus.

Reinforcement learning algorithms for real-world robotic applications must be able to handle complex, unknown dynamical systems while maintaining data-efficient learning. These requirements are handled well by model-free and model-based RL approaches, respectively. In this work, we aim to combine the advantages of these approaches. By focusing on time-varying linear-Gaussian policies, we enable a model-based algorithm based on the linear-quadratic regulator that can be integrated into the model-free framework of path integral policy improvement. We can further combine our method with guided policy search to train arbitrary parameterized policies such as deep neural networks. Our simulation and real-world experiments demonstrate that this method can solve challenging manipulation tasks with comparable or better performance than model-free methods while maintaining the sample efficiency of model-based methods.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

Citations in the corpus

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Methods based on local models, such as guided policy search algorithms [Levine et al., 2016, Finn et al., 2016, Chebotar et al., 2017], can efficiently train NN policies, but use time-varying linear models, which only locally model the system dynamics.”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Deep neural network models have been used widely in a range of robotics applications [12, 13, 14, 15, 16, 17].”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “While reinforcement learning (RL) has achieved some success in domains such as assembly [1], ping pong [2], in-hand manipulation [3], and hockey [4], state-of-the-art methods require substantially more experience than humans to acquire only one narrowly-defined skill.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “…approaches addressing the planning component of MBRL including linear controllers for local models (Levine et al., 2016; Kumar et al., 2016; Chebotar et al., 2017), uncertainty-aware (Chua et al., 2018b; Gal et al., 2016) or deterministic planners (Nagabandi et al., 2018) and stochastic…”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, methodology): “…ongoing for decades, with techniques such as synthetic experience generation (Sutton, 1990), partial model-based backpropagation (Nguyen & Widrow, 1990; Heess et al., 2015), and layering model-free learning on the residuals of model-based estimation (Chebotar et al., 2017) being a few examples.”
Exploring Model-based Planning with Policy Networks (2019, background): “Many previous attempts have tried to distill the planned trajectories into a policy network [21, 23, 4, 43], and control only with policy network.”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “Model-based RL methods have achieved signiﬁcant efﬁciency beneﬁts compared to model-free RL methods [9, 30, 10].”
Deep reinforcement learning: a survey (2020, methodology): “To adopt good points from both model-free and model-based RL algorithms, Chebotar et al. (2017) focused on time-varying linear-Gaussian policies, and integrated a model-based LQR algorithm with a model-free path integral policy improvement algorithm.”
Differentiable MPC for End-to-end Planning and Control (2018, unknown): “…be generated by model-based control algorithms to guide the policy search or prime a model-free policy [Sutton, 1990, Theodorou et al., 2010, Levine and Abbeel, 2014, Gu et al., 2016b, Venkatraman et al., 2016, Levine et al., 2016, Chebotar et al., 2017, Nagabandi et al., 2017, Sun et al., 2017].”
Automated Curriculum Learning by Rewarding Temporally Rare Events (2018, background): “achieved impressive results learning to play video games [17] and controlling both simulated and physical robots [2, 9, 15, 27].”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, methodology): “Prior works have considered instrumenting the environment with additional sensors to evaluate rewards (Gu et al., 2017; Chebotar et al., 2017; Zhu et al., 2019), which is a highly manual process, using demonstrations, which require manual effort to collect (Vecerik et al.”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, methodology): “TCN originally used PILQR for their RL algorithm [34], though any RL algorithm can be used.”
Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods (2018, unknown): “For example, guided policy search methods have been applied for solving a range of manipulation tasks, including contact-rich, visionbased skills [23], non-prehensile manipulation [10], and tasks involving significant discontinuities [5], [4].”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, methodology): “In each iteration i, PILQR performs a KL-constrained optimization of S(xt,ut): min p(i) Ep(i) [S(xt,ut)] s.t. Ep(i−1) [ DKL ( p(i)‖ p(i−1) )] ≤ , 2We use a cost function instead of the reward function as it is more common in the trajectory optimization theory. where limiting the KL-divergence betwee”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “This can be through human orchestrated resets [8, 13, 12, 4, 16], which requires high frequency human intervention in robot training.”
Deep Reinforcement Learning: An Overview (2017, unknown): “, 2016) • issue: model-based learning proposed approaches: – Dyna-Q (Sutton, 1990) – combine model-free and model-based RL (Chebotar et al., 2017) • issue: model-free planning proposed approaches: – value iteration networks (Tamar et al.”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “In the scenario where the dynamics are unknown, LQRbased methods are typically used in a “trajectory-centric” fashion where the distributions over initial conditions and goal conditions are low variance (Levine & Abbeel, 2014; Chebotar et al., 2017).”