Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction

By Mohit Shridhar, David Hsu. 2018

161 citations recorded by the source corpus.

This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions. INGRESS allows for unconstrained object categories and unconstrained language expressions. Further, it asks questions to disambiguate referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural network model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expression, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred object. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans.

Subfields: LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In LLM Agents & Tool Use

Citations in the corpus

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Instruction following is a long-standing topic of interest in robotics [7, 10, 29, 34, 39, 40, 46, 51].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “While language-grounding for manipulation has been explored in the past [7, 8, 9, 10], these pipelines are limited by object-centric representations that cannot handle granular or deformable objects and often do not reason about perception and action in an integrated manner.”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, methodology): “Following the common evaluation [58, 59], we report two metrics.”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, background): “ing interactive fetching systems to localize objects mentioned in referring expressions [17], [18] or by grounding not only objects, but also spatial relations to follow language expressions characterizing pick-and-place commands [19]–[21].”
Robots That Use Language (2020, methodology): “They have been implemented in physical robots in an object retrieval setting by training the robot on simulated images (110, 111).”
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks (2021, background): “Recent successes in humanrobot interaction include an interactive fetching system to localize objects mentioned in referring expressions [12], [13], [14], [15], [16] or grounding not only objects, but also spatial relations to follow language expressions characterizing pick-and-place commands [17], ”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It is useful in a broad range of tasks, such as language-conditioned manipulation [9, 10, 11, 12, 13, 14, 15], visual language navigation [16, 17], and visual question answering [18, 19, 20, 21, 22, 23, 24].”
A Joint Network for Grasp Detection Conditioned on Natural Language Commands (2021, background): “ambiguous commands need clarification through humanrobot dialogue [7], [8].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Our language-conditioned segmentation problem is a variant of referring expression recognition [31, 32, 33, 34, 35].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”