TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

By Howard Chen, Alane Suhr, Dipendra Kumar Misra, Noah Snavely, Yoav Artzi. 2018

471 citations recorded by the source corpus.

We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task. We introduce the Touchdown task and dataset, where an agent must first follow navigation instructions in a Street View environment to a goal position, and then guess a location in its observed environment described in natural language to find a hidden object. The data contains 9326 examples of English instructions and spatial descriptions paired with demonstrations. We perform qualitative linguistic analysis, and show that the data displays a rich use of spatial reasoning. Empirical analysis shows the data presents an open challenge to existing methods.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, background): “…navigation tests the agent’s ability to take action according to human instructions, which recently gains popularity in embodied AI (Anderson et al., 2018b; Chen et al., 2019; Jain et al., 2019; Chen et al., 2019; Qi et al., 2020b; Krantz et al., 2020; Nguyen and Daumé III, 2019; Ku et al., 2020).”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “However, our findings are also relevant for similar datasets such as Touchdown (Chen et al., 2019; Mehta et al., 2020), CVDN (Thomason et al., 2019), REVERIE (Qi et al., 2020), and the multilingual Room-acrossRoom (RxR) dataset (Ku et al., 2020).”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “Learning to navigate in a photorealistic home environment based on natural language instructions has attracted increasing research interest [23, 14, 7, 3, 6], as it provides insight into core scientific questions about multimodal representations.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Human-generated natu-ral language instructions are used in [ MacMahon et al. , 2006; Bisk et al. , 2016; Misra et al. , 2017; Janner et al. , 2018; Chen et al. , 2018; Anderson et al. , 2018; Goyal et al. , 2019; Wang et al. , 2019 ] .”
Vision-and-Dialog Navigation (2019, background): “Dataset —Language Context— —Visual Context— Human Amb UnderS Temporal Real-world Temporal Shared MARCO[8, 9], DRIF[10] 3 7 7 1I 7 Dynamic R2R[7], Touchdown[11] 3 7 7 1I 3 Dynamic EQA[12], IQA[13] 7 7 3 1Q 7 Dynamic CLEVR[14] 7 7 1Q 7 Static VQA[15, 16, 17] 3 7 1Q 3 Static CLEVR-Dialog[18] 7 7 2D 7 S”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, methodology): “, 2020) and outdoor (Chen et al., 2019; Mehta et al., 2020) environments; success in these is based on clearly-defined, objective task completion rather than language or vision specific annotations.”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “The richness and generalizability of natural language makes it an effective medium for directing mobile agents in navigation tasks, even in environments they have never encountered before (Anderson et al., 2018b; Chen et al., 2019; Misra et al., 2018; de Vries et al., 2018; Qi et al., 2019).”
REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments (2019, background): “…QA-pair – Dynamic QA MARCO [21], DRIF [3] Nav-Instruction Detailed Dynamic Navigation R2R [1] Nav-Instruction Detailed Dynamic Navigation TouchDown [5] Nav-Instruction Detailed Dynamic Navigation VLNA [24], HANNA[23] Nav-Dialog High Dynamic Find Object TtW [8] Nav-Dialog High Dynamic Navigation…”
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, methodology): “Simulators that are more photo-realistic, realistic-physics enabled simulators are beginning to be utilized to train realworld embodied agents [53, 10, 13, 59, 21, 40].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, unknown): “To incorporate linguistic knowledge into driving, a few datasets are proposed to benchmark outdoor grounding and visual language navigation tasks [155], [156], [157], [158].”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “Spatial reasoning [162] refers to instructions that may include the spatial relationship of the object or part to be grasped, necessitating the inference of grasping posture based on the spatial relationships of objects within the scene.”
ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, background): “Embodied question answering (EQA) [17, 44, 47] is the combination of VQA and vision-and-language navigation task [5, 11, 41, 45].”
Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View (2020, methodology): “Our Retouchdown reimplementation of the RC ONCAT model improves over the results given in Chen et al. (2019) for all metrics.”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “Visual navigation can be tied to other tasks to attain intelligent behavior, such as question answering [36,22,23], active visual recognition [48], and instruction following [5,15].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “Future Work As suggested by the significant improvement on R2R [3] achieved by our VLNœ BERT, we expect that the model can improve the performance of other navigation settings such as street navigation [6] and navigation in continuous environments [22].”
Executing Instructions in Situated Collaborative Interactions (2019, background): “Executing single instructions has been widely studied (e.g., Tellex et al., 2011; Duvallet et al., 2013; Misra et al., 2017, 2018; Anderson et al., 2018; Blukis et al., 2018a,b; Chen et al., 2019).”
VALAN: Vision and Language Agent Navigation (2019, methodology): “The framework facilitates the development and evaluation of embodied agents for solving grounded language understanding tasks, such as Vision-and-Language Navigation and Vision-andDialog Navigation, in photo-realistic environments, such as Matterport3D [7] and Google StreetView [8].”