Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

By Bill Byrne, K. Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, A. Cedilnik, Kyu-Young Kim. 2019

246 citations recorded by the source corpus.

A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken “Wizard of Oz” (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is “self-dialog” in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Scaling Instruction-Finetuned Language Models (2022, background): “(2021) and 26 new tasks that we added in this work, including dialog data (Byrne et al., 2019; Anantha et al., 2021; Dai et al., 2022) and program synthesis data (Yasunaga and Liang, 2020; Li et al.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “We additionally add a number of task-oriented dialogue tasks: GoogleSGD (Rastogi et al., 2020) and Taskmaster 1, 2 & 3 (Byrne et al., 2019).”
Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “For example, the restaurant domain appears in TM19, TM20 and MWOZ, with different slot names and values.”
SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model (2020, methodology): “We pre-train the grounded response generation model on dialog corpora (Kim et al., 2019; Rastogi et al., 2019; Byrne et al., 2019; Eric and Manning, 2017; Mrkšić et al., 2016; Asri et al., 2017)8 as shown”
Dialog Inpainting: Turning Documents into Dialogs (2022, methodology): “Datasets We use four open-domain conversational QA retrieval benchmarks: OR-QuAC (Qu et al., 2020), TREC CAsT-19 (Byrne et al., 2019), TREC CAsT-20 (Dalton et al., 2020), and QReCC (Anantha et al., 2021).”
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling (2021, background): “Similarly, Rastongi et al.[1, 30] applied M2M to collect a large-scale schema-guided ToD dataset, and Kottur et al.[31] extended it to multimodal setting.”
MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines (2020, background): “, 2018), TaskMaster (Byrne et al., 2019), SGD (Rastogi et al.”
RyanSpeech: A Corpus for Conversational Text-to-Speech Synthesis (2021, background): “Taskmaster-2: This dataset is also designed for use in goaloriented dialogue systems [16].”
Open-Domain Question Answering Goes Conversational via Question Rewriting (2020, methodology): “This design decision is called selfdialog technique that was shown to help improve quality of the data by avoiding some of the challenges observed in simulated dialogues produced by pairs of annotators (Byrne et al., 2019).”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, unknown): “Similar to PPTOD, our DIAL-BART0 model is first pre-trained on 7 datasets: KVRET (Eric et al., 2017), WOZ (Mrkši´c et al., 2017), Cam-Rest676 (Wen et al., 2017), MSR-E2E (Li et al., 2018), Frames (El Asri et al., 2017), TaskMaster (Byrne et al., 2019), Schema-Guided (Rastogi et al., 2020b) along wit”
Task-Oriented Dialogue as Dataflow Synthesis (2020, background): “Existing ﬁne-grained dialogue act schemes (Stolcke et al., 2000) can distinguish among a range of communicative intents not modeled by our approach, and slot-ﬁlling representations have historically been easier to predict (Zue et al., 1994) and annotate (Byrne et al., 2019).”
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2020, methodology): “The TicketTalk movie ticketing dataset was created using the self-dialog collection method (Krause et al., 2017; Moghe et al., 2018; Byrne et al., 2019) in which a paid crowd-sourced worker writes both sides of the dialog (i.e. both customer and ticketing agent turns) based on a particular scenario…”
Simulated Chats for Task-oriented Dialog: Learning to Generate Conversations from Instructions (2020, background): “…chat logs for intents and dialog acts (Yu and Yu 2019; Rastogi et al. 2020), create dialog summaries (Gliwa et al. 2019), converse with each other based on a script (Zhang et al. 2018a) or converse to accomplish tasks or goals (El Asri et al. 2017; Budzianowski et al. 2018; Byrne et al. 2019) etc.”
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020, background): “Taskmaster (Byrne et al., 2019) 13,215 303,066 22.”
Dialogue Transformers (2019, methodology): “[26] Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik.”
TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog (2020, methodology): “These datasets were primarily collected through Wizard-of-Oz (WoZ) and self-written dialog (i.e., where one person is writing an imagined dialog), through a crowdsourcing platform: • The Taskmaster (Byrne et al., 2019) dataset (self-written and WoZ using a TTS).”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “However, we believe that it has a huge potential in addressing other tasks which require sequential interactions/decisions, such as language and visual dialog [5, 8, 21, 25, 45], dialog navigation [9, 40, 41] and action anticipation for reactive robot response [1, 13, 20].”