Prospection: Interpretable plans from language by predicting the future

By Chris Paxton, Yonatan Bisk, Jesse Thomason, Arunkumar Byravan, Dieter Fox. 2019

51 citations recorded by the source corpus.

High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various works have investigated strategies for the challenging problem of fusing vision, language, and control [60, 61, 62, 63, 64, 65, 66].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47].”
Language Grounding with 3D Objects (2021, background): “By predicting the new information that can be gotten from different actions, agents can prospect over potential futures for planning [40, 41, 42].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Prior, representation learning was studied on language tasks restricted to simulated discrete [10, 11, 12, 13, 14, 15] and continuous [16, 17, 18, 19] environments, or non-language robotic tasks [20, 21, 22, 23].”
Correcting Robot Plans with Natural Language Feedback (2022, background): “2) Language for Robot Task Speciﬁcation: Natural language has been extensively studied as a means for task speci-ﬁcation or instruction following in robotics [37, 22, 15, 4, 28, 43, 8, 31].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, background): “[25] generate subgoals from natural language descriptions of the task, which are then used by a controller to predict actions.”
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, background): “Modern approaches use neural architectures to map natural language to the (simulated) world and execute actions (Paxton et al., 2019; Chen et al., 2018; Nguyen et al., 2018; Blukis et al., 2018; Fried et al., 2018; Mei et al., 2016).”