Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions

By Peter Alexander Jansen. 2020

62 citations recorded by the source corpus.

The recently proposed ALFRED challenge task aims for a virtual robotic agent to complete complex multi-step everyday tasks in a virtual home environment from high-level natural language directives, such as “put a hot piece of bread on a plate”. Currently, the best-performing models are able to complete less than 1% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language directives into detailed multi-step sequences of actions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to generate gold multi-step plans from language directives alone without any visual input in 26% of unseen cases. When a small amount of visual information, the starting location in the virtual environment, is incorporated, our best-performing GPT-2 model successfully generates gold command sequences in 58% of cases, suggesting contextualized language models may provide strong planning modules for grounded virtual agents.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. Pieter Abbeel
4. Karol Hausman
5. Igor Mordatch
6. Andy Zeng
7. Li Fei-Fei
8. Trevor Darrell

In Computer Vision and Pattern Recognition

1. Sergey Levine
2. Luc Van Gool
3. Li Fei-Fei
4. Trevor Darrell
5. Jitendra Malik
6. Dacheng Tao
7. Xiaogang Wang
8. Abhinav Kumar Gupta

In Large Language Models

1. Andy Zeng
2. Brian Ichter
3. Sergey Levine
4. Fei Xia
5. Karol Hausman
6. Dieter Fox
7. Pieter Abbeel
8. Igor Mordatch

In Natural Language Processing

1. Dieter Fox
2. Valts Blukis
3. Brian Ichter
4. Jesse Thomason
5. Fei Xia
6. Pieter Abbeel
7. Sergey Levine
8. Animesh Garg

Citations in the corpus

• Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various prior works have explored using language as a space for planning [51, 52, 53, 20, 54, 21].”
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Perhaps more importantly, few works have evaluated LLMs in an embodiment setting that realizes the full potential of the actionable knowledge these models already contain by pre-training on large-scale unstructured text: the tasks evaluated are often generated from a handful of templates, which do n”
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “…agents [19–22] and of robotics [23, 6, 24, 25, 5, 7, 26], where language can be used as a tool for compositional goal specification [5, 27–33], semantic anchor for training multi-modal representation [12, 34, 35], or as an intermediate substrate for planning and reasoning [36– 38, 9, 10, 39, 40].”
• Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021, unknown): “Previous work (Jansen, 2020) models sub-goal planning”
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2022, background): “Prompting LLMs to generate text useful for robot task planning is a nascent topic [32], [33], [34], [2], [4],”