Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

By Nasrin Mostafazadeh, Chris Brockett, William B. Dolan, Michel Galley, Jianfeng Gao, Georgios P. Spithourakis, Lucy Vanderwende. 2017

193 citations recorded by the source corpus.

The popularity of image sharing on social media and the engagement it creates between users reflect the important role that visual context plays in everyday conversations. We present a novel task, Image Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialog research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jason Aaron Edward Weston
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Yejin Choi
  5. Percy Liang
  6. Luke S. Zettlemoyer
  7. Yoshua Bengio
  8. Zhe Gan

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Chunhua Shen
  5. Xiaogang Wang
  6. Cordelia Schmid
  7. Ming-Hsuan Yang
  8. Jitendra Malik

In Multimodal Learning

  1. Zhe Gan
  2. Linjie Li
  3. Qi Wu
  4. Jianfeng Gao
  5. Jingjing Liu
  6. Anton van den Hengel
  7. Devi Parikh
  8. Yu Cheng

In Dialogue & Conversational AI

  1. Jason Aaron Edward Weston
  2. Kurt Shuster
  3. Minlie Huang
  4. Can Xu
  5. Eric Michael Smith
  6. Stephen Roller
  7. Xiaoyan Zhu
  8. Dongyan Zhao

Citations in the corpus