Learning symmetric and low-energy locomotion

By Wenhao Yu, Greg Turk, C. Karen Liu. 2018

223 citations recorded by the source corpus.

Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function [Duan et al. 2016]. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually look nothing like a real runner. This paper takes a minimalist learning approach to the locomotion problem, without the use of motion examples, finite state machines, or morphology-specific knowledge. We introduce two modifications to the DRL approach that, when used together, produce locomotion behaviors that are symmetric, low-energy, and much closer to that of a real person. First, we introduce a new term to the loss function (not the reward function) that encourages symmetric actions. Second, we introduce a new curriculum learning method that provides modulated physical assistance to help the character with left/right balance and forward movement. The algorithm automatically computes appropriate assistance to the character and gradually relaxes this assistance, so that eventually the character learns to move entirely without help. Because our method does not make use of motion capture data, it can be applied to a variety of character morphologies. We demonstrate locomotion controllers for the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our results show that learned policies are able to produce symmetric, low-energy gaits. In addition, speed-appropriate gait patterns emerge without any guidance from motion examples or contact planning.

Subfields: Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Learning quadrupedal locomotion over challenging terrain (2020, neutral): “Instead of directly using the reward function to evaluate the learning progress [25, 41–43], we evaluate cTs by the traversability of generated terrains, which is defined as the success rate of traversing a terrain.”
Embodied intelligence via learning and evolution (2021, methodology): “We measure energy as the absolute sum of all joint torques 55 .”
Learning Locomotion Skills for Cassie: Iterative Design and Sim-to-Real (2019, methodology): “More recently, DRL has been applied to 3D bipedal locomotion, e.g., [25, 26, 27].”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “In fact, most of the prior work used simulation (Brockman et al. 2016; Coumans and Bai 2016) to evaluate and benchmark the learning algorithms (H ¨ am ¨ al ¨ ainen et al. 2015; Heess et al. 2017; Yu et al. 2018; Peng et al. 2018a).”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Deep RL also has been used to learn locomotion control policies [29, 52, 43, 9, 55], mostly in simulated environments.”
AMP (2021, methodology): “Heuristics such as symmetry, stability, and effort minimization can improve the realism of motions produced by physically simulated characters [Grochow et al. 2004; Mordatch et al. 2012, 2013; Yu et al. 2018].”