MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation
By Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, Qingwei Lin. 2022
80 citations recorded by the source corpus.
Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog dataset to facilitate multi-modal conversation better. MMDialog is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 88x. Second, it contains massive topics to generalize the open domain. To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios. In addition, we build two baselines for the above tasks with state-of-the-art techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset is available in https://github.com/victorsungo/MMDialog.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Eric P. Xing
- Mohit Bansal
- Ion Stoica
- Adyasha Maharana
- Joseph E. Gonzalez
- Zhanghao Wu
- Zi Lin
- Haotong Zhang
In Computer Vision and Pattern Recognition
- Eric P. Xing
- Luc Van Gool
- Mohit Bansal
- Trevor Darrell
- Dacheng Tao
- Andrew Zisserman
- Ming-Hsuan Yang
- Jitendra Malik
In Natural Language Processing
- Adyasha Maharana
- Eric P. Xing
- Mohit Bansal
- Zhanghao Wu
- Haotong Zhang
- Sergey Tulyakov
- Lianmin Zheng
- Francesco Barbieri
In Multimodal Learning
Citations in the corpus
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “• Conversational benchmarks, like CoQA [34], MMDialog [15] and OpenAssistant [22], are closest to our intended use cases.”
- Evaluating Very Long-Term Conversational Memory of LLM Agents (2024, methodology): “We assess such consistency by measuring how closely the predicted multi-modal dialogues align with the ground truth multi-modal dialogues in our dataset, quantifying this alignment through MMRelevance (Feng et al., 2022), in addition to other NLG metrics.”