Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following

By Valts Blukis, Ross A. Knepper, Yoav Artzi. 2020

37 citations recorded by the source corpus.

We study the problem of learning a robot policy to follow natural language instructions that can be easily extended to reason about new objects. We introduce a few-shot language-conditioned object grounding method trained from augmented reality data that uses exemplars to identify objects and align them to their mentions in instructions. We present a learned map representation that encodes object locations and their instructed use, and construct it from our few-shot grounding output. We integrate this mapping approach into an instruction-following policy, thereby allowing it to reason about previously unseen objects at test-time by simply adding exemplars. We evaluate on the task of learning to map raw observations and instructions to continuous control of a physical quadcopter. Our approach significantly outperforms the prior state of the art in the presence of new objects, even when the prior approach observes all objects during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various works have investigated strategies for the challenging problem of fusing vision, language, and control [60, 61, 62, 63, 64, 65, 66].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “Leveraging language embed-dings overcomes the limitations of learned task-embeddings by distilling natural language into semantically rich embeddings; however, prior work in mapping language to task abstractions has primarily considered navigation problems [8], [9], [10], [11], [12].”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “Language grounding has been studied extensively both in terms of intelligent agents [19–22] and of robotics [23, 6, 24, 25, 5, 7, 26], where language can be used as a tool for compositional goal specification [5, 27–33], semantic anchor for training multimodal representation [12, 34, 35], or as an i”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, background): “Modelling improvements have enabled these approaches to achieve good performance on complex navigation tasks [7, 9, 11, 13, 14, 37], a success that has not yet translated to mobile manipulation [12, 46, 47].”
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models (2021, unknown): “To alleviate reliance on human annotation, some works have investigated zero-/few-shot grounding of new object types (Sadhu et al., 2019; Blukis et al., 2020), whereas amounts of training data are still needed for existing object types.”