Talk the Walk: Navigating New York City through Grounded Dialogue
By Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Aaron Edward Weston, Douwe Kiela. 2018
130 citations recorded by the source corpus.
We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Luke S. Zettlemoyer
- Percy Liang
- Jason Aaron Edward Weston
- Caiming Xiong
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
In Experimental and Cognitive Psychology
- Jason Baldridge
- Ali Farhadi
- Jason Aaron Edward Weston
- Yejin Choi
- Hal Daumé Iii
- Douwe Kiela
- Siddhartha S. Srinivasa
- Jianfeng Gao
In Dialogue & Conversational AI
- Jason Aaron Edward Weston
- Alexander Gerhard Schwing
- Jianfeng Gao
- Marimuthu Kalimuthu
- Hal Daumé Iii
- Caiming Xiong
- Dietrich Klakow
- Samuel Humeau
Citations in the corpus
- Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “Talk the Walk (de Vries et al., 2018) two-way dialogue, but is limited to small grids.”
- Vision-and-Dialog Navigation (2019, background): “Dataset —Language Context— —Visual Context— Human Amb UnderS Temporal Real-world Temporal Shared MARCO[8, 9], DRIF[10] 3 7 7 1I 7 Dynamic R2R[7], Touchdown[11] 3 7 7 1I 3 Dynamic EQA[12], IQA[13] 7 7 3 1Q 7 Dynamic CLEVR[14] 7 7 1Q 7 Static VQA[15, 16, 17] 3 7 1Q 3 Static CLEVR-Dialog[18] 7 7 2D 7 S”
- TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, background): “Our environment includes 29 , 641 panoramas and 61 , 319 edges from New York City.”
- Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “In future work, we would like to incorporate our model on navigation tasks involving dialogue (Thoma-son et al., 2019; de Vries et al., 2018) and real robot platforms (Banerjee et al., 2020) where life-long learning is possible (Thomason et al., 2015; Johansen et al., 2020).”
- Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “The richness and generalizability of natural language makes it an effective medium for directing mobile agents in navigation tasks, even in environments they have never encountered before (Anderson et al., 2018b; Chen et al., 2019; Misra et al., 2018; de Vries et al., 2018; Qi et al., 2019).”
- Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…(Das et al., 2018; Gordon et al., 2018), and navigation (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Zhu et al., 2017; de Vries et al., 2018; Yuke Zhu, 2017; Mousavian et al., 2018; Wayne et al., 2018; Wang et al., 2018a; Mirowski et al., 2017; 2018; Zamir et al., 2018).”
- Tactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation (2019, background): “Due to the enormous visual complexity of real world scenes, the literature surrounding VLN also builds off of computer vision work in referring expressions [15, 24], visual question answering [3], and most recently ego-centric QA that requires navigation to answer questions [11, 8, 9].”
- OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs (2019, background): “…recommendations (Salem et al., 2014; Bordes et al., 2017; Sun and Zhang, 2018; Dalton, 2018), information querying (Williams et al., 2017; de Vries et al., 2018; Reddy et al., 2018), etc., with datasets collected mostly through bootstrapped simulations (Bordes et al., 2017), Wizard-of-Oz…”
- Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, background): “Simulators that are more photo-realistic, realistic-physics enabled simulators are beginning to be utilized to train realworld embodied agents [53, 10, 13, 59, 21, 40].”
- The Regretful Agent: Heuristic-Aided Navigation Through Progress Estimation (2019, methodology): “This includes House3D [22], Embodied QA [7], AI2-THOR [12], navigation based agents [15, 20, 14] (including with communication [9]), and the VLN task that we focus on [2].”
- A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Embodied navigation also aids AI agents to develop behavior such as instruction following [38,4,82,95,3], city navigation [18,64,63,94], question answering [21,22,34,97,24], and active visual recognition [105,104].”
- Transferable Representation Learning in Vision-and-Language Navigation (2019, background): “ content, their alignment and the agent’s position in the environment. VLN datasets have graduated from simple virtual environments [26] to photo-realistic environments, both indoors [2] and outdoors [10, 7, 19]. To succeed, VLN agents must internalize the (possibly noisy) natural language instructi”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Other applications include automatic surveillance (Baumann et al., 2008), autonomous driving (Kim et al., 2018), humancomputer interaction (Rickert et al., 2007), city navigation (de Vries et al., 2018) and so on.”
- Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View (2020, methodology): “Regions containing PII were marked as bounding boxes by annotators, and we blurred all of these regions for the final images.”
- Executing Instructions in Situated Collaborative Interactions (2019, unknown): “…various scenarios, including dialogue where only one side acts in the world (Anderson et al., 1991; Williams et al., 2013; Vlachos and Clark, 2014; de Vries et al., 2018; Kim et al., 2019; Hu et al., 2019), coordination for agreed selection of an object (He et al., 2017; Udagawa and Aizawa,…”
- VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “However, we believe that it has a huge potential in addressing other tasks which require sequential interactions/decisions, such as language and visual dialog [5, 8, 21, 25, 45], dialog navigation [9, 40, 41] and action anticipation for reactive robot response [1, 13, 20].”