Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

By Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, Yoav Artzi. 2018

200 citations recorded by the source corpus.

We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “The CHAI benchmark [32] evaluates agents performing household instructions, but uses a generic “interact” action.”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[56] Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “…and Parikh, 2013; Antol et al., 2015), caption generation (Chen et al., 2016), referring expression resolution (Mitchell et al., 2010; Matuszek et al., 2012; FitzGerald et al., 2013), and instruction following (MacMahon et al., 2006; Chen and Mooney, 2011; Bisk et al., 2016; Misra et al., 2018).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “Attempts to integrate deep RL with natural language typically employ less realistic grid-like environments [Misra et al., 2017] or grant agents privileged global observations and gold-standard action sequences to make learning more tractable [Misra et al., 2018].”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, methodology): “The most related resources to ours are R2R [2] and Talk the Walk [10].”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, neutral): “, 2020) and CHAI (Misra et al., 2018).”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “…the ability to request additional help while executing tasks (Misra et al., 2014, 2017; Anderson et al., 2018b; Chen et al., 2019; Das et al., 2018; Misra et al., 2018; Wijmans et al., 2019; Qi et al., 2019), or mimic human verbal assistance with primitive, highly scripted language (Nguyen et al.,…”
AllenAct: A Framework for Embodied AI Research (2020, background): “AI2-THOR [3], CHAI [45], and Virtual Home [4] are examples of platforms that go beyond navigation and enable evaluation of agents on tasks that require interaction such as applying forces and/or changing object states.”
Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following (2019, methodology): “In this work, we use instruction following [16, 17, 25, 23, 18] as a natural testbed to examine this question.”
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (2020, methodology): “[20] introduce a simulated environment with unconstrained navigation and a dataset of crowdsourced instructions; however, the environments are unrealistic, synthetic scenes.”
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021, unknown): “To facilitate task learning from natural language instructions, several benchmarks using simulated physical environment have been made available (Anderson et al., 2018; Misra et al., 2018; Blukis et al., 2019; Shridhar et al., 2020).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, unknown): “Also, in the context of interpretable learning framework learning-by-asking (LBA) (Misra et al., 2018b) is proposed to closely mimic natural learning with the goal to make it more data efficient than the traditional VQA setting.”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, background): “where EGOTRANSFORM(x, Pt) transforms the map x to the agent egocentric pose Pt, REFINER is a neural network based on the LingUNet architecture [36], and φ is the language embedding.”
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (2019, background): “…either simulated (Macmahon et al., 2006; Yan et al., 2018; Bisk et al., 2018; Shah et al., 2018), simulations of realistic (Blukis et al., 2018; Misra et al., 2018) and real environments (Anderson et al., 2018b; de Vries et al., 2018; Chen et al., 2019; Cirik et al., 2018), or actual physical…”
Shifting the Baseline: Single Modality Performance on Visual Navigation & QA (2018, background): “The task can be further generalized by requiring an agent to navigate and interact with the world to answer a question (e.g., (Misra et al., 2018)).”
Building Cooperative Embodied Agents Modularly with Large Language Models (2023, unknown): “…al., 2022; Wang et al., 2023d), especially in embodied environments (Li et al., 2023a; Padmakumar et al., 2022; Kolve et al., 2017; Shridhar et al., 2020; Misra et al., 2018; Zhu et al., 2017; Brodeur et al., 2017; Xia et al., 2018; Savva et al., 2019; Xiang et al., 2020; Jain et al., 2020; 2019).”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “Vision-Language Navigation deﬁnes the task of requiring an agent to navigate through a visual environment based on natural language instructions (Anderson et al., 2018b; Misra et al., 2018; Chen et al., 2019; Jain et al., 2019; Nguyen and Daumé III, 2019; Thomason et al., 2020).”