Learning agile and dynamic motor skills for legged robots
By Jemin Hwangbo, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, Marco Hutter. 2019
1,799 citations recorded by the source corpus.
A method for learning agile control policies uses simulated data to enable precise, efficient movements in a complex physical robot. Legged robots pose one of the greatest challenges in robotics. Dynamic and agile maneuvers of animals cannot be imitated by existing methods that are crafted by humans. A compelling alternative is reinforcement learning, which requires minimal craftsmanship and promotes the natural evolution of a control policy. However, so far, reinforcement learning research for legged robots is mainly limited to simulation, and only few and comparably simple examples have been deployed on real systems. The primary reason is that training with real robots, particularly with dynamically balancing systems, is complicated and expensive. In the present work, we introduce a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system, thereby leveraging fast, automated, and cost-effective data generation schemes. The approach is applied to the ANYmal robot, a sophisticated medium-dog–sized quadrupedal system. Using policies trained in simulation, the quadrupedal machine achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.
Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Yoshua Bengio
- Ruslan Salakhutdinov
- Trevor Darrell
- Mohit Bansal
- Quoc V. Le
In Biomedical Engineering
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Gabriel B. Margolis
- Jitendra Malik
- Abhinav Kumar Gupta
- Vladlen Koltun
- Jie Tan
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhinav Kumar Gupta
- Nicolas Manfred Otto Heess
- Lerrel Pinto
- Vikash Kumar
- Dieter Fox
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shixiang Shane Gu
- Nicolas Manfred Otto Heess
- Ofir Nachum
- Abhinav Kumar Gupta
- Shimon Whiteson
In Robotics & Control
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhinav Kumar Gupta
- Lerrel Pinto
- Dieter Fox
- Dhruv Batra
- Vladlen Koltun
Citations in the corpus
- Solving Rubik's Cube with a Robot Hand (2019, methodology): “[48] used real data to learn an actuation model and combined it with domain randomization to successfully transfer locomotion policies.”
- SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “One potential approach to enable a robot to perform these manipulation tasks is with deep reinforcement learning (RL), which has achieved many successes in recent years [4, 5, 6, 7, 8].”
- Learning quadrupedal locomotion over challenging terrain (2020, positive): “This information allows the policy to exploit the actuator dynamics (12).”
- RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “…on-the-fly, meaning that you cam generate an infinite amount for your imitation learning algorithm. c) Sim-to-Real Transfer : Recently there has been a large amount of work in learning control policies in simulation and then transferring these to the real world [48], [49], [50], [51], [52], [53].”
- Habitat: A Platform for Embodied AI Research (2019, background): “Once a promising approach has been developed and tested in simulation, it can be transferred to physical platforms that operate in the real world [6, 15].”
- Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control (2019, background): “Most documented legged robots’ Froude numbers are smaller than 2 [2], [16], except Cheetah 2 demonstrated 6.”
- Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior (2022, methodology): “The tradeoffs between model-based control and reinforcement learning are well discussed in prior literature [3, 22].”
- RVT: Robotic View Transformer for 3D Object Manipulation (2023, background): “The learning of robotic control policy has been traditionally studied with low-dimensional state observations [9, 10, 11, 12, 13].”
- Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (2021, methodology): “For this reason and following the methodology of previous work [1], we use a neural network to compute torques from joint position commands.”
- Learning robust perceptive locomotion for quadrupedal robots in the wild (2022, methodology): “One curriculum adjusts terrain difficulty using an adaptive method [4] and the other changes elements such as reward or applied disturbances using a logistic function [3].”
- Robot Parkour Learning (2023, background): “Recently, learning-based methods have shown robust performance on walking [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 12, 21, 22, 23, 24, 25, 26, 27], climbing stairs [20, 28, 29, 30, 31, 32, 33], mimicking animals [34, 35, 36, 37, 38, 39] and legged mobile manipulation [40, 41, 42] by…”
- How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “Other than domain randomization, other parallel research directions for sim-to-real transfer exist and have been demonstrated to be promising research areas as well ([21], [22], [23], [24], [11]).”
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, methodology): “…models, either analytical (28) or neural network-based (29), from robot data to improve simulation fidelity; 2) randomizing dynamics parameters (28, 29) and, even further, randomizing morphology (30), which enables generalization to unseen quadrupeds; and 3) adopting a hierarchi-6 cal structure…”
- Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, background): “Hwangbo et al., 2019; Molchanov et al., 2019; Andrychowicz et al., 2020).”
- Real-world humanoid locomotion with reinforcement learning (2023, background): “For example, learning-based approaches have proven very e ff ective in dexterous manipulation ( 13 – 15 ), quadrupedal locomotion ( 16 – 18 ), and bipedal locomotion ( 19 – 23 ).”
- Sim2Real Transfer for Deep Reinforcement Learning with Stochastic State Transition Delays (2020, background): “Deep Reinforcement Learning (RL) has shown promising results for a range of robotics applications, such as navigation [1], manipulation [2], and locomotion [3].”
- RMA: Rapid Motor Adaptation for Legged Robots (2021, methodology): “Alternately, the Sim-to-Real gap can also be reduced by making the simulation more accurate [23, 51, 19].”
- Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Although researchers have developed principled approaches for mitigating the sim-toreal issue, including system identification [14, 31], domain randomization [45, 50, 47], and meta-learning [24, 53, 56], it remains an open problem.”