Image-Chat: Engaging Grounded Conversations

By Kurt Shuster, Samuel Humeau, Antoine Bordes, Jason Aaron Edward Weston. 2018

131 citations recorded by the source corpus.

To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the Image-Chat test set (preferred 47.7% of the time).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Empirical evidence suggests that data with a very large range of domains (e.g., social media data) provides the best basis for fine-tuning (Zhang et al., 2019; Shuster et al., 2020; Adiwardana et al., 2020; Roller et al., 2020).”
Deep Learning for Text Style Transfer: A Survey (2020, background): “A useful downstream application of TST is persona-consistent dialog generation (Li et al. 2016; Zhang et al. 2018b; Shuster et al. 2020).”
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, methodology): “Shuster et al. (2020) released Image-Chat that is larger than IGC and consists of 202K image-grounded dialogues.”
All-in-One Image-Grounded Conversational Agents (2019, methodology): “Image Chat (IC) Image Chat [Shuster et al., 2018] is an extension of the personality captions dataset to full dialogue.”
Word meaning in minds and machines (2020, background): “…grounded in the ways discussed in the ﬁ rst desideratum, but new efforts have focused on more grounded forms of dialog through curriculum-driven language learning (Mikolov et al., 2016), text-based adventure games (Urbanek et al., 2019),and discussions of natural images (Shuster et al., 2020).”
Learning to Ground Visual Objects for Visual Dialog (2021, background): “Recent several works (Shuster et al., 2018; Liang et al., 2021; Yang et al., 2020) explore leveraging visual information to enhance dialogue modLTMI LTMI-LG”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Visual Dialogue; Speaker Consistency and Personality Response Shuster et al. (2018)”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, background): “For example, for the multimodal open-domain dialogue dataset Image-Chat (Shuster et al., 2020), each data example includes an image; the dialogue history between two speakers A and B; speaker style traits.”
Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, background): “The image-grounded dialogue task is centered around responding to questions (Antol et al., 2015; Das et al., 2017; Kottur et al., 2019) or creating natural conversations related to specific images (Mostafazadeh et al., 2017; Shuster et al., 2018; Meng et al., 2020; Zheng et al., 2021).”
Multi-Modal Open-Domain Dialogue (2020, methodology): “Further still, there have been recent forays into open-domain natural dialogue in the context of images, e.g. in the Image-Chat (Shuster et al., 2020) and Image-grounded Conversations (Mostafazadeh et al., 2017) tasks.”