Learning to Walk via Deep Reinforcement Learning

By Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, George Tucker, Sergey Levine. 2018

501 citations recorded by the source corpus.

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website.

Subfields: Artificial Intelligence, Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

Citations in the corpus

Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “Our implementation of PreQN was signiﬁcantly slower than our implementation of SAC (by more than 50%), due to the requirement of calculating backward passes separately for each state-action pair in the batch, and solving the system of equations K θ Z = ∆ .”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “With the progress to overcome these challenges, we have developed an efﬁcient and autonomous on-robot training system (Haarnoja et al. 2019), in which the robot can learn walking and turning, from scratch in the real world, with only ﬁve minutes of data (Yang et al. 2020) and little human…”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “Empirically, MaxEnt RL algorithms achieve good performance on a number of simulated [17] and real-world [16, 46] control tasks, and can be more robust to perturbations [18].”
RMA: Rapid Motor Adaptation for Legged Robots (2021, background): “…require considerable expertise on the part of the human designer, and in recent years there has been much interest in replicating this success using reinforcement learning and imitation learning techniques [23, 18, 41, 55, 32] which could lower this burden, and perhaps also improve performance.”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “This can be through human orchestrated resets [8, 13, 12, 4, 16], which requires high frequency human intervention in robot training.”
Learning Agile Robotic Locomotion Skills by Imitating Animals (2020, background): “Reinforcement learning has been effective for automatically acquiring locomotion skills in simulation [44, 34, 32] and in the real world [31, 59, 14, 58, 22, 26].”
Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real (2019, background): “an outdoor natural terrain) [2] and in the presence of complex or faulty hardware.”
Data Efficient Reinforcement Learning for Legged Robots (2019, methodology): “5b), or 1.6 body lengths per second, which is twice the fastest speed achieved by [10].”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “robots [26, 39] or relied on tedious manual resets between roll-outs [28, 16].”
Robot Parkour Learning (2023, background): “Recently, learning-based methods have shown robust performance on walking [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 12, 21, 22, 23, 24, 25, 26, 27], climbing stairs [20, 28, 29, 30, 31, 32, 33], mimicking animals [34, 35, 36, 37, 38, 39] and legged mobile manipulation [40, 41, 42] by…”