The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents

By Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, Jason Aaron Edward Weston. 2019

87 citations recorded by the source corpus.

We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pre-trained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the fine-tune and task transfer settings. We obtain state-of-the-art results on many of the tasks, providing a strong baseline for this challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Empirical evidence suggests that data with a very large range of domains (e.g., social media data) provides the best basis for fine-tuning (Zhang et al., 2019; Shuster et al., 2020; Adiwardana et al., 2020; Roller et al., 2020).”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “The model was pre-trained on 128 V100 GPUs for approximately 25 days.”
Measuring Attribution in Natural Language Generation Models (2021, neutral): “…that paper, we also compared with three models that are seq2seq-style conversation models: the original answer generation system from Dinan et al. (2019), the Dodecadialogue multitask system from Shuster et al. (2020) and a T5-base model (Raffel et al., 2020) finetuned on Wizard of Wikipedia data.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “In addition it uses Pushshift.io Reddit, a variant of Reddit discussions, which has also been used in several existing studies (see e.g., Yang et al. (2018); Mazaré et al. (2018); Shuster et al. (2020)).”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “For the 90M parameter model, results are reported from Shuster et al. (2019), as we use that same model.”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “A natural setting for exploring these ideas might be dialogue (Hancock et al., 2019; Shuster et al., 2020).”
Q^{2}: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021, methodology): “We evaluate the Image+Seq2Seq dodecaDialogue multi-task model fine-tuned on WOW (Shuster et al., 2020),”
Multi-Modal Open-Domain Dialogue (2020, methodology): “…Transformer sequence-to-sequence model trained on 341GB of conversations (Adiwardana et al., 2020) • The Image+Seq2Seq model from dodecaDia-logue (Shuster et al., 2019b), a Transformer sequence-to-sequence model in which the encoder is passed pre-trained image features from the ResNeXt-IG-3.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Shuster et al. (2020) collected a dataset (IMAGE-CHAT, shown in Figure 17) of image-grounded human-human conversations in which speakers are asked to perform role-playing based on an emotional mood or style offered, since the usage of such characteristics is also a significant factor in…”
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features (2021, methodology): “We also include results the end-to-end generative model (E2E) with gold knowledge that was introduced in the original Wizard of Wikipedia paper (Dinan et al., 2019) and the model in the follow-up work on dodecaDialogue (Shuster et al., 2020).”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “V&L Multi-Task Learning Instead of building a monolithic model for different V&L tasks, numbers of previous work explore multi-task learning with a unified model for utilising the common and the complementary knowledge to reduce the domain gap [31, 39, 42, 47, 33].”
DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue (2020, background): “Within conversational systems, the recently proposed Dialogue Dodecathlon is a benchmark for knowledge-grounded, situated, and multi-modal dialogue generation consisting of several opendomain dialogue datasets (Shuster et al., 2019).”
All-in-One Image-Grounded Conversational Agents (2019, methodology): “…of skills, a large set of imageand-text tasks have been devised by researchers, covering image captioning [Young et al., 2014; Chen et al., 2015; Shuster et al., 2019a], visual question answering [Goyal et al., 2017; Das et al., 2017], and dialogue based on an image [Mostafazadeh et al., 2017;…”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “• Safe is the test set from Blended Skill Talk (BST) [56].”