Talk the Walk: Navigating New York City through Grounded Dialogue

By Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Aaron Edward Weston, Douwe Kiela. 2018

130 citations recorded by the source corpus.

We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yejin Choi
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Jason Aaron Edward Weston
  7. Caiming Xiong
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Cordelia Schmid
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Ming-Hsuan Yang
  8. Andrew Zisserman

In Experimental and Cognitive Psychology

  1. Jason Baldridge
  2. Ali Farhadi
  3. Jason Aaron Edward Weston
  4. Yejin Choi
  5. Hal Daumé Iii
  6. Douwe Kiela
  7. Siddhartha S. Srinivasa
  8. Jianfeng Gao

In Dialogue & Conversational AI

  1. Jason Aaron Edward Weston
  2. Alexander Gerhard Schwing
  3. Jianfeng Gao
  4. Marimuthu Kalimuthu
  5. Hal Daumé Iii
  6. Caiming Xiong
  7. Dietrich Klakow
  8. Samuel Humeau

Citations in the corpus