XIRL: Cross-embodiment Inverse Reinforcement Learning

By Kevin Zakka, Andy Zeng, Peter R. Florence, Jonathan Tompson, Jeannette Bohg, Debidatta Dwibedi. 2021

160 citations recorded by the source corpus.

We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments -- shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings typically required alignment with a reference trajectory, which may be difficult to acquire under stark embodiment differences. We show empirically that if the embeddings are aware of task progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. Additionally, when transferring real-world human demonstrations to a simulated robot, we find that XIRL is more sample efficient than current best methods. Qualitative results, code, and datasets are available at https://x-irl.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, weights associated with one but not other modalities, frozen for downstream tasks (Zhai et al., 2021; Florence et al., 2019; Tsimpoukelli et al., 2021; Zakka et al., 2022).”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “Most prior works exploit the idea of learning an invariant representation between human and robot domains to transfer the demonstrated skills (Sermanet et al., 2016; 2018; Schmeckpeper et al., 2020; Chen et al., 2021; Xiong et al., 2021; Zakka et al., 2022; Bahl et al., 2022).”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Similarly, Zakka et al. [105] took a goal-driven approach, concentrating on mimicking the task progress without the need for detailed structural matching.”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “…to address the embodiment gap between different robots, such as shared action representations [14, 30], incorporating representation learning objectives [17, 26], adapting the learned policy on embodiment information [11, 15, 18, 30, 31], and decoupling robot and environment representations [24].”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Several prior works have explored using human video data in robot learning, for example to acquire goals [41, 42, 43], to learn visual dynamics models [44, 45, 46, 47], or to learn representations and rewards [19, 48, 49, 50, 51, 52].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “LOReL [81] crowd-sources humans labels to train language-conditioned reward function for ofﬂine RL. AVID [113] and XIRL [142] extract reward signals via cycle consistency.”
Visual Language Maps for Robot Navigation (2022, background): “, cross-embodiment problem [36], [37]), which may be useful for multi-agent coordination [38].”
Code as Policies: Language Model Programs for Embodied Control (2022, background): “CaP exhibit a degree of cross-embodiment support [59], [60] by performing the same task differently depending on the action APIs.”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, background): “A plethora of recent research has explored leveraging largescale human video data to improve robot policy learning [10, 12, 13, 42, 43, 46, 47, 48, 53, 58, 60].”
Human Hands as Probes for Interactive Object Understanding (2021, background): “For instance, distance in a state-sensitive feature space can be used as reward functions for manipulation tasks [52,54,64].”