Learning locomotion skills using DeepRL: does the choice of action space matter?

By Xue Bin Peng, Michiel van de Panne. 2016

201 citations recorded by the source corpus.

The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts learning and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gait-cycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and motion quality of the resulting policies.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “When designing the action space, we choose to use the position control mode of the actuators for safety and ease of learning [36].”
Learning agile and dynamic motor skills for legged robots (2019, background): “[57] found that such a controller can outperform a torque controller in both training speed and final control performance.”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “We use the target joint angles of proportional derivative (PD) controllers as the action representation, which enables robust motion imitation as observed in prior work [60, 92].”
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks (2019, background): “[33] studied the importance of different action representations in RL for the task of locomotion.”
Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning (2019, background): “Peng and van de Panne [18] showed that using this type of action representations for learning motor skills achieves better final performance, faster learning, and higher robustness compared to other representations such as target joint torques and target joint velocities.”
Data Efficient Reinforcement Learning for Legged Robots (2019, background): “Whereas formulating the action space using desired motor angles is easier to learn [24], controlling the motors in position control mode can result in abrupt changes in desired motor angles, which may cause large torque output that could potentially damage the robot and its surroundings.”