Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning

By N. Rudin, David Hoeller, Philipp Reist, Marco Hutter. Conference on Robot Learning, 2021

1,008 citations recorded by the source corpus.

In this work, we present and study a training set-up that achieves fast policy generation for real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy performance and training times. In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal to walk on challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup of multiple orders of magnitude compared to previous work. Finally, we transfer the policies to the real robot to validate the approach. We open-source our training code to help accelerate further research in the field of learned legged locomotion.

Subfields: Artificial Intelligence, Biomedical Engineering, Software

View Paper: https://www.semanticscholar.org/paper/ca6096142016a2ba8133f6cb2c04ad30f5eae730

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Software

Citations in the corpus

Legged Locomotion in Challenging Terrains using Egocentric Vision (2022, methodology): “Recent methods skip 248 foothold planning and directly train a deep RL policy that takes the elevation map as input and outputs 249 either low-level motor primitives [9, 70] or raw joint angles [14, 71, 72, 73].”
Extreme Parkour with Legged Robots (2023, neutral): “…results set a new landmark in learning-driven park-our with high jumps that are 2x the height of the robot, long jumps that are 2x the length of the robot, walking on front two legs (handstand), and jumping over titled ramps directly from a single front-facing egocentric depth camera (see Table I).”
Scaling the Number of Tasks in Continual Learning (2022, background): “The need for the sequence of tasks or episodes to converge would explain why reinforcement learning trainings are time and data-consuming [Rafﬁn et al., 2022, Rudin et al., 2022, Gupta et al., 2022].”
Rapid locomotion via reinforcement learning (2022, methodology): “Recent works have successfully employed RL to learn locomotion controllers [23, 24, 27, 29, 35, 37].”
Robot Parkour Learning (2023, background): “[29, 61, 31], elevation maps [28, 109, 110, 111, 112], lidar scans [113], RGB images [32], event cameras [68] or learned neural spaces [30, 33], but none have demonstrated effective high-speed agile locomotion.”
Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior (2022, methodology): “To overcome these issues, we leverage human intuition about useful behavior parameters (bt) corresponding to gait properties like foot swing motion, body posture, and contact schedule [6, 7, 8, 23].”
ANYmal parkour: Learning agile navigation for quadrupedal robots (2023, methodology): “a) Walking: The robot must traverse various irregular terrains consisting of stairs, slopes, and randomly placed small obstacles, similar to the ones commonly used in previous legged locomotion works [7], [41].”
Masked Visual Pre-training for Motor Control (2022, methodology): “Rudin et al. (2021) have shown that sim-to-real transfer is feasible based on policies trained in IsaacGym.”
DayDreamer: World Models for Physical Robot Learning (2022, background): “…approach is to train RL agents from large amounts of simulated data under domain and dynamics randomization (Peng et al., 2018; Lee et al., 2020; Rudin et al., 2021; Siekmann et al., 2021; Escontrela et al., 2022; Miki et al., 2022; Kumar et al., 2021; Rusu et al., 2016; Bohez et al., 2022),…”