Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real

By Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Shane Gu, Vikash Kumar. 2019

101 citations recorded by the source corpus.

Manipulation and locomotion are closely related problems that are often studied in isolation. In this work, we study the problem of coordinating multiple mobile agents to exhibit manipulation behaviors using a reinforcement learning (RL) approach. Our method hinges on the use of hierarchical sim2real -- a simulated environment is used to learn low-level goal-reaching skills, which are then used as the action space for a high-level RL controller, also trained in simulation. The full hierarchical policy is then transferred to the real world in a zero-shot fashion. The application of domain randomization during training enables the learned behaviors to generalize to real-world settings, while the use of hierarchy provides a modular paradigm for learning and transferring increasingly complex behaviors. We evaluate our method on a number of real-world tasks, including coordinated object manipulation in a multi-agent setting. See videos at this https URL

Subfields: Artificial Intelligence, Control and Systems Engineering, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Social Psychology

Citations in the corpus

AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “Applying this change of variables to (10) (after some telescoping, see Nachum et al. (2019b)) yields max The resulting objective is now completely oﬀ-policy, relying only on access to samples from the initial state distribution µ 0 and the oﬀ-policy dataset d D .”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “…(Silver et al., 2016; Tesauro, 1995; Brown & Sandholm, 2019) and robotics (Levine et al., 2016; Dasari et al., 2019; Peters et al., 2010; Parmas et al., 2019; Pinto & Gupta, 2016; Nachum et al., 2019a), where the agent can act in the environment and sample as many transitions and rewards as needed.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “Reinforcement learning (RL) algorithms have recently demonstrated impressive success in learning behaviors for a variety of sequential decision-making tasks (Barth-Maron et al., 2018; Hessel et al., 2018; Nachum et al., 2019).”
Imitation Learning via Off-Policy Distribution Matching (2019, methodology): “First, Nachum et al. (2019b) only use the divergence formulation for data distribution estimation (which is used for off-policy evaluation), assuming a ﬁxed policy.”
Legged Locomotion in Challenging Terrains using Egocentric Vision (2022, methodology): “This has motivated work in using RL for learning to walk in 237 simulation [11, 29, 30, 31], and then successfully deploy them in a diverse set of real world scenarios 238 [32, 33, 34, 35, 36, 37, 32, 38].”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Nachum et al. (190) focus on enabling multiple quadrupeds to perform complex tasks like manipulation and coordination using model-free RL.”
Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? (2019, background): “…options framework (Precup, 2000) and goal-conditioned hierarchies (Nachum et al., 2018b), due to their impressive success in recent work (Frans et al., 2018; Levy et al., 2017; Nachum et al., 2018a; 2019), though an examination of other architectures is an important direction for future research.”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, methodology): “P USH , inspired from Nachum et al. (2019), simulated in the MuJoCo (Todorov et al., 2012) physics engine.”
RMA: Rapid Motor Adaptation for Legged Robots (2021, background): “Domain randomization is a class of methods in which the policy is trained with a wide range of environment parameters and sensor noises to learn behaviours which are robust in this range [51, 52, 40, 54, 38].”
Intrinsic Motivation for Encouraging Synergistic Behavior (2020, background): “Furthermore, we consider a multi-agent locomotion task: ant push (loosely inspired by a domain considered by Nachum et al. (2019)).”
Toward Robust Long Range Policy Transfer (2021, background): “Although domain randomization (Tobin et al. 2017; Nachum et al. 2020) in the pre-training phase may mitigate this problem, we prefer the pre-trained policies to gradually fit the transferring tasks.”