Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks

By Roberto Mart'in-Mart'in, Michelle A. Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, Animesh Garg. 2019

238 citations recorded by the source corpus.

Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for variable impedance control in end-effector space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

robosuite: A Modular Simulation Framework and Benchmark for Robot Learning (2020, methodology): “The controllers span the most frequently used action spaces (joint space and Cartesian space) and have been tested and used in previous projects [11].”
Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning (2021, background): “OSC [11] is a task-space compliant controller that has been shown to enable faster policy learning compared to joint-space controllers [33] and learn contact-rich tasks [16].”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, background): “In related work [84], we have looked into also modulating the force applied to the environment by using variable impedance control on task such as surface wiping and door opening.”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “These methods often introduce mechanisms specifically designed to address the embodiment gap between different robots, such as shared action representations [14, 30], incorporating representation learning objectives [17, 26], adapting the learned policy on embodiment information [11, 15, 18, 30, 31]”
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments (2021, unknown): “…kinds of embodied AI methods, here we evaluate two reinforcement learning (RL) algorithms that have demonstrated excellent results in simpler embodied AI tasks with continuous or discrete action spaces [61, 62, 21, 63–67]: Soft-Actor Critic (SAC [16]) and Proximal-Policy Optimization (PPO [17]).”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Some studies have attempted to benchmark different action spaces (66, 86, 199), but such principled studies and guidelines are still lacking for many problems.”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, background): “Employing these controllers, particularly in task space, has shown to help sim-to-real transferability of robot manipulation policies [7], [35].”