Topological Planning with Transformers for Vision-and-Language Navigation

By Kevin Chen, Junshen Chen, Jo Chuang, Marynel Vázquez, Silvio Savarese. 2020

174 citations recorded by the source corpus.

Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. FORWARD, ROTATE) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigation plans, and exhibits intelligent behaviors such as backtracking.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “While mapping environments during inference has shown promise on both VLN (Fang et al., 2019; Chen et al., 2021) and ALFRED (Blukis et al.”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “A number of studies [5, 10, 7, 57] have presented compelling methodologies that underscore the significance of topological maps in facilitating long-term planning, specifically in the aspect of backtracking to prior locations.”
Multimodal Learning With Transformers: A Survey (2022, unknown): “vision-and-language navigation (VLN) [68], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83]), communication (e.”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “While these approaches have been largely studied in simulated spatial games [19, 23, 24, 25] or in object-directed visual navigation in simulated robots [26, 27, 28, 29, 30, 31, 25] some of which include high-level object interaction [32], in this work we focus on the domain of learning control for ”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, background): “Recently, topological maps [17], [18] and scene graphs [19], [20] emerge as an effective discrete representation of scenes.”