Affordances from Human Videos as a Versatile Representation for Robotics

By Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Deepak Pathak. Computer Vision and Pattern Recognition, 2023

337 citations recorded by the source corpus.

Building a robot that can understand and learn to interact by watching humans has inspired several vision problems. However, despite some successful results on static datasets, it remains unclear how current models can be used on a robot directly. In this paper, we aim to bridge this gap by leveraging videos of human interactions in an environment centric manner. Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show how to seamlessly integrate our affordance model with four robot learning paradigms including offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. We show the efficacy of our approach, which we call Vision-Robotics Bridge (VRB) across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/253b41369d003952874c6a47a6038277b165cfa0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “…Anything Model (TAM) [62] Open-Vocabulary 3D Scene Representation e.g. CLIP-NERF [63], LERF [64], DFF [65] Affordances e.g. Affordance Diffusion [66], VRB [67] Embodied AI e.g. Huang et al. [68], Statler [69], EmbodiedGPT [70], MineDojo [71], VPT [72], Kwon et al. [16], Voyager [73], ELLM [74] Fig.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “Recently, vision-language-action (VLA) models have garnered signiﬁcant attention for their ability to extend pre-trained vision-languagemodelstoroboticsusinganext-tokenprediction approach.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, neutral): “The learned cost functions are then used by reinforcement learning [102, 99, 107], imitation learning [97, 96], or trajectory optimization [95, 35] to generate robot actions.”
Octo: An Open-Source Generalist Robot Policy (2024, background): “Multiple works leverage diverse non-robot data or pretrained vision-language foundation models to boost policy generalization to new scenes and tasks [86, 103, 96, 16, 38, 11, 84, 36, 4, 37, 7, 3, 46, 15, 23].”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, background): “Utilizing passive human demonstrations, previous works learn task cost functions [37, 8, 1, 21], affordance functions [2], dense object descriptors [40, 24, 39], action correspondences [33, 28], and pre-trained visual representations [23, 48].”