Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?

By Yuchen Cui, Scott Niekum, Abhi Gupta, Vikash Kumar, Aravind Rajeswaran. 2022

102 citations recorded by the source corpus.

Task specification is at the core of programming autonomous robots. A low-effort modality for task specification is critical for engagement of non-expert end-users and ultimate adoption of personalized robot agents. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. The former is hard to interpret for non-experts and necessitates detailed state estimation and scene understanding. The latter requires the generation of desired goal image, which often requires a human to complete the task, defeating the purpose of having autonomous robots. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use such as images obtained from the internet, hand sketches that provide a visual description of the desired task, or simple language descriptions. As a preliminary step towards this, we investigate the capabilities of large scale pre-trained models (foundation models) for zero-shot goal specification, and find promising results in a collection of simulated robot manipulation tasks and real-world datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Engineered success detectors can operate on ground-truth state in simulation, while learned success detectors can be trained on real examples of successes and failures in the real world [78, 79, 80, 81, 82].”
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, methodology): “These visually-grounded models have been used for several downstream tasks, such as in robotic success detection [18, 20, 54, 65], action prediction [7, 56], and reward prediction [16, 22, 46, 48].”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, methodology): “To enable generalization in the wild, a parallel line of works has explored learning task specification from large-scale offline data [104–106, 35, 34, 44, 107, 108, 54], particularly egocentric videos [109, 110], or leveraging pre-trained foundation models [111–113, 33, 114, 115].”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “…covering a wide range of pre-existing visual representations that have been used for robotics control (Shah & Kumar, 2021; Parisi et al., 2022; Cui et al., 2022), though none has been tested in our three reward-based settings in which the reward also has to be produced by the representation.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Prior works have explored the use of natural language in robot manipulation, primarily as a means of task speciﬁcation [53, 54, 36, 55] or reward learning [56].”