Data Efficient Reinforcement Learning for Legged Robots

By Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, Vikas Sindhwani. 2019

169 citations recorded by the source corpus.

We present a model-based framework for robot locomotion that achieves walking based on only 4.5 minutes (45,000 control steps) of data collected on a quadruped robot. To accurately model the robot's dynamics over a long horizon, we introduce a loss function that tracks the model's prediction over multiple timesteps. We adapt model predictive control to account for planning latency, which allows the learned model to be used for real time control. Additionally, to ensure safe exploration during model learning, we embed prior knowledge of leg trajectories into the action space. The resulting system achieves fast and robust locomotion. Unlike model-free methods, which optimize for a particular task, our planner can use the same learned dynamics for various tasks, simply by changing the reward function. To the best of our knowledge, our approach is more than an order of magnitude more sample efficient than current model-free methods.

Subfields: Artificial Intelligence, Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

Citations in the corpus

WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Robot learning has emerged as a strong paradigm for automatically acquiring complex and skilled behaviors such as locomotion (Yang et al., 2019; Peng et al., 2020), navigation (Mirowski et al., 2017; Kahn et al., 2020), and manipulation (Gu et al., 2017; et al, 2019).”
Model-Based Offline Planning (2020, unknown): “[23] show good results learning legged locomotion policies using MPC with learned models, and [24] demonstrate flexible robot arm controllers leveraging learned models with image-based goals.”
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, background): “Although real-robot learning has been achieved with fully-observable MBRL [6, 7, 8, 9, 10, 11], there has been an increasing demand for robot learning in partially observable environments in which only incomplete information (especially vision) is available.”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, methodology): “Taking the robotic control as an example (Nagabandi et al., 2018a; Yang et al., 2020; Rakelly et al., 2019; Gu et al., 2017; Bousmalis et al., 2018; Raileanu et al., 2020; Yang et al., 2019), dynamics change caused by parts damages could easily lead to the failure of MBRL algorithms.”
Learning to Walk in the Real World with Minimal Human Effort (2020, methodology): “Otherwise, the learning process requires a large number of manual resets between roll-outs [28, 16, 54], which limits the scalability of the learning system.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “…et al. 2018; Yang et al. 2020), have scaled to vision-based tasks (Levine et al. 2016; Finn et al. 2016b; Finn and Levine 2017), and demonstrated generalization capabilities to many objects and tasks when the model is trained on large, diverse datasets (Finn and Levine 2017; Yang et al. 2020).”
RMA: Rapid Motor Adaptation for Legged Robots (2021, background): “One approach is to directly train in the real world [18, 53].”
DayDreamer: World Models for Physical Robot Learning (2022, methodology): “A model based263 algorithm first learns a dynamics model, which can then be used to plan actions, or be used as a264 simulator to learn a model free policy, (Nagabandi et al., 2019; Hafner et al., 2018; Yang et al., 2019;265 Hafner et al., 2019; 2020).”