Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

By Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, Vincent Vanhoucke. 2018

981 citations recorded by the source corpus.

Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer to the real world. We narrow this reality gap by improving the physics simulator and learning robust policies. We improve the simulation using system identification, developing an accurate actuator model and simulating latency. We learn robust controllers by randomizing the physical environments, adding perturbations and designing a compact observation space. We evaluate our system on two agile locomotion gaits: trotting and galloping. After learning in simulation, a quadruped robot can successfully perform both gaits in the real world.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…and autonomous driving, a rapidly growing research topic is the study of simulation to real-world transfer: training policies with reinforcement learning in simulation and then transferring these policies into the real world (Sadeghi and Levine, 2017; Tan et al., 2018; Chebotar et al., 2019).”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “This idea has also been extended to dynamics randomization [80, 3, 105, 119, 77] to learn a robust policy that transfers to similar environments but with different dynamics.”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, background): “Combination of system identification and dynamics randomization has been used in the past to learn locomotion for a real quadruped [26], non-prehensile object manipulation [27] and in-hand object pivoting [28].”
Learning quadrupedal locomotion over challenging terrain (2020, critical): “In our formulation, unlike many existing works [12, 14, 16] that focus on tracking the target velocity of the base ( BIB v T ), only the direction ( BIB ˆ v T ) is given to the controller.”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “This approach has been employed for manipulation [24], visual servoing [25], locomotion [26], and navigation [23].”
Learning dexterous in-hand manipulation (2018, methodology): “This idea has also been extended to dynamics randomization [4, 61, 68] to learn a robust policy that transfers to similar environments but with different dynamics.”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “The action space are the swing angle and the extension of each leg, which are then mapped to desired motor positions and tracked with a PD controller (Tan et al., 2018).”
A Distributional View on Multi-Objective Policy Optimization (2020, background): “…it possible to train policies that exceed human-level performance on Atari (Mnih et al., 2015) and Go (Silver et al., 2016), perform complex robotic manipulation tasks (Zeng et al., 2019), learn agile locomotion (Tan et al., 2018), and even obtain reward in unanticipated ways (Amodei et al., 2016).”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, methodology): “The interface between the DRL policy and the model-based controller could be defined at various levels, such as joint positions (31, 32, 33), leg poses (28), gait parameters (34, 35), or temporally-extended macro actions (36).”
First return, then explore (2020, background): “…of training trials current RL algorithms require, as well as the safety concerns that arise when running RL directly in the real world, simulators have played a key role in training the most compelling applications of RL 8,22–24 , and will likely continue to be harnessed for the foreseeable future.”
Adaptive Power System Emergency Control Using Deep Reinforcement Learning (2019, background): “One of the authors has made good progress in addressing this reality gap issue in the robotic domain [41].”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “Other than domain randomization, other parallel research directions for sim-to-real transfer exist and have been demonstrated to be promising research areas as well ([21], [22], [23], [24], [11]).”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, methodology): “While simulators have been used in robotics to help alleviate this difficulty, such as for locomotion [59] and rigid object manipulation [47], many simulated benchmarks for manipulation focus on rigid objects, partially because of difficulties in simulation of deformables [4].”
Learning Agile Robotic Locomotion Skills by Imitating Animals (2020, background): “Sim-toreal transfer can be facilitated by constructing more accurate simulations [58, 62], or adapting the simulator with realworld data [57, 23, 26, 36, 5].”
Data Efficient Reinforcement Learning for Legged Robots (2019, methodology): “Recently, they have enabled automation of the design process for locomotion controllers [2, 5, 6, 7, 8].”