MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling

By Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gasic. 2018

1,527 citations recorded by the source corpus.

Even though machine learning has become the major scene in dialogue research community, the real breakthrough has been blocked by the scale of data available.To address this fundamental obstacle, we introduce the Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics.At a size of 10k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora.The contribution of this work apart from the open-sourced dataset is two-fold:firstly, a detailed description of the data collection procedure along with a summary of data structure and analysis is provided. The proposed data-collection pipeline is entirely based on crowd-sourcing without the need of hiring professional annotators;secondly, a set of benchmark results of belief tracking, dialogue act and response generation is reported, which shows the usability of the data and sets a baseline for future studies.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “Similar reductions in errors were seen in task-oriented dialog as a result of improving the dataset (Budzianowski et al., 2018).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “…2020), which is a held-out datato-text task for transforming data triples to text, Spider (Yu et al., 2018), a SQL query generation task given a database and an input query, and fully supervised in our framework, and MultiWoZ (Budzianowski et al., 2018), is a held-out dialogue state tracking task.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Traditional goal-oriented dialogue has focused on narrow tasks that would typically be useful for a dialogue-based assistant, for example restaurant (Henderson, Thomson, and Williams, 2014), taxi, train, and hotel (Budzianowski et al., 2018) or trip (El Asri et al., 2017) booking.”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “• Conversational tasks require understanding of not only the user’s last request but also the full interaction history between users and machines (Budzianowski et al., 2018; Eric et al., 2019; Yu et al., 2019a).”
Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, methodology): “We take five different tasks from the MultiWOZ dataset (Budzianowski et al., 2018) which are hotel, restaurant, train, taxi and attraction.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “In this case, we further incorporate Wik-iSQL (SQL query generation, Zhong et al., 2017), CNN/DailyMail (news article summarization See et al., 2017), MultiWOZ (semantic state sequence generation (Budzianowski et al., 2018)) into our task sequences 1 .”
Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking (2021, neutral): “…dialogues for training/validation/testing, belonging to calendar scheduling, weather query, and POI navigation domains, (2) Multi-WOZ 2.1 (Budzianowski et al., 2018) that contains 1,839/117/141 dialogues for training/validation/testing, belonging to restaurant, attraction, and hotel…”
Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “Consequently, in this paper we propose a Continual Learning benchmark by jointly pre-processing four task-oriented dataset such as Task-Master 2019 (TM19) (Byrne et al., 2019), Task-Master 2020 (TM20) (Byrne et al., 2019), Schema Guided Dialogue (SGD) (Rastogi et al., 2019) and MultiWoZ (Budzianowsk”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, methodology): “0 dataset 2 (Budzianowski et al., 2018) containing six domains ( Attraction, Hotel, Restaurant, Booking, Taxi and Train ) and seven DA intents ( “Inform, Request, Select, Rec-ommend, Book, Offer-Booked, No-Offer” ).”
Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, positive): “In our experiments, we have taken K =0 and K =1 to create the dataset for intent detection from SGD and MWOZ datasets.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation (2022, neutral): “0 dataset (Budzianowski et al., 2018).”
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset (2019, methodology): “0 (Wen et al. 2017), FRAMES (El Asri et al. 2017), M2M (Shah et al. 2018) and MultiWOZ (Budzianowski et al. 2018).”
Few-shot Natural Language Generation for Task-Oriented Dialog (2020, methodology): “To simulate such a few-shot learning setting, we have developed a new benchmark dataset, called FEWSHOTWOZ, based on the MultiWOZ (Budzianowski et al., 2018) and Cambridge NLG datasets (Wen et al.”
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (2019, methodology): “Text task (see Figure 1) on the multi-domain MultiWOZ dataset (Budzianowski et al., 2018b) suggest that our GPT-based task-oriented dialogue model learns to generate and understand domain-specific tokens, which in turn leads to a seamless adapta-”
Text-to-Text Pre-Training for Data-to-Text Tasks (2020, background): “• MultiWoz (Budzianowski et al., 2018) is a corpus of 10K human-human dialogs for developing task oriented dialogue systems.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Most traditional conversational datasets [6, 10, 31, 71] are solely text based, and notoriously difficult to collect, relying on narrowly constructed ontologies [31, 55] and highly specific domains [6, 10].”
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems (2019, background): “One of the motivations to perform unseen domain DST is because collecting a large-scale task-oriented dataset for a new domain is expensive and time-consuming (Budzianowski et al., 2018), and there are a large amount of domains in realistic scenarios.”
Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (2019, methodology): “Size: The total of 13,215 dialogs in this corpus is on par with similar, recently released datasets such as MultiWOZ (Budzianowski et al., 2018).”