Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
By Nasrin Mostafazadeh, Chris Brockett, William B. Dolan, Michel Galley, Jianfeng Gao, Georgios P. Spithourakis, Lucy Vanderwende. 2017
193 citations recorded by the source corpus.
The popularity of image sharing on social media and the engagement it creates between users reflect the important role that visual context plays in everyday conversations. We present a novel task, Image Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialog research.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Dialogue & Conversational AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jason Aaron Edward Weston
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Zhe Gan
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Chunhua Shen
- Xiaogang Wang
- Cordelia Schmid
- Ming-Hsuan Yang
- Jitendra Malik
In Multimodal Learning
In Dialogue & Conversational AI
- Jason Aaron Edward Weston
- Kurt Shuster
- Minlie Huang
- Can Xu
- Eric Michael Smith
- Stephen Roller
- Xiaoyan Zhu
- Dongyan Zhao
Citations in the corpus
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “, 2018) and Image Grounded Conversations (Mostafazadeh et al., 2017).”
- Neural Approaches to Conversational AI (2018, neutral): “At a high level, most of these works have in common the idea of augmenting their context encoder to not only represent the conversation history, but also some additional input drawn from the user’s environment, such as an image (Das et al., 2017a; Mostafazadeh et al., 2017) or textual information (G”
- Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “There are also setups that consider static language and perception, for example image captioning (Lin et al., 2014), video captioning (Yu et al., 2016), visual QA (Antol et al., 2015) and visual dialogue (Das et al., 2017; Shuster et al., 2018; Mostafazadeh et al., 2017).”
- The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, background): “It can also discuss images, as this is a vital part of human connection (Image Chat, IGC).”
- Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “Only recently have researchers begun to explore how to ground open-domain dialog systems in real-world entities and knowledge [36, 88, 104].”
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “Other related work includes imagegrounded chitchat (Mostafazadeh et al., 2017), dialog-based image retrieval (Guo et al.”
- Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning (2017, background): “…al., 2017), visual question answering (Antol et al., 2015; Fukui et al., 2016; Lu et al., 2016; Zhu et al., 2017), visual story-telling (Huang et al., 2016), natural question generation (Mostafazadeh et al., 2017, 2016), and image generation from captions (Mansimov et al., 2016; Reed et al., 2016).”
- Image-Chat: Engaging Grounded Conversations (2018, methodology): “Automatic Evaluation In (Mostafazadeh et al., 2017), the authors provide BLEU scores for their models in an attempt to evaluate their effectiveness via automated metrics.”
- Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Visual Dialogue Mostafazadeh et al. (2017) Image-Chat Contain conversations grounded with images.”
- Answer-based Adversarial Training for Generating Clarification Questions (2019, background): “Mostafazadeh et al. (2016, 2017) generate natural and engaging questions, given an image (and some initial text).”
- MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, methodology): “Firstly, several corpora, including Visual Dialog, IGC and Image-Chat, are derived from crowd-sourcing dialogues talking about given images.”
- Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, background): “More recent work has explored the role of image as context for interaction and pragmatics, either in dialog (Mostafazadeh et al., 2016, 2017), or as a prompt for users to generate descriptions (Bisk et al., 2019).”
- Are You Talking to Me? Reasoned Visual Dialog Generation Through Adversarial Learning (2017, background): “[20] propose an Image Grounded Conversation (IGC) dataset and task that requires a model to generate natural-sounding conversations (both questions and responses) about a shared image.”
- Knowledge-Grounded Dialogue Generation with Pre-trained Language Models (2020, background): “, 2020), or extracted from visual background (Mostafazadeh et al., 2017; Shuster et al., 2018; Huber et al., 2018).”
- Engaging Image Chat: Modeling Personality in Grounded Dialogue (2018, background): “Perhaps the most related work to ours is (Mostafazadeh et al., 2017).”
- Low-Resource Knowledge-Grounded Dialogue Generation (2020, background): “, 2018), and images (Mostafazadeh et al., 2017), has emerged as a new fashion in the research of open domain dialogue systems.”