Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset

By Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, Pranav Khaitan. 2019

732 citations recorded by the source corpus.

Virtual assistants such as Google Assistant, Alexa and Siri provide a conversational interface to a large number of services and APIs spanning multiple domains. Such systems need to support an ever-increasing number of services with possibly overlapping functionality. Furthermore, some of these services have little to no training data available. Existing public datasets for task-oriented dialogue do not sufficiently capture these challenges since they cover few domains and assume a single static ontology per domain. In this work, we introduce the the Schema-Guided Dialogue (SGD) dataset, containing over 16k multi-domain conversations spanning 16 domains. Our dataset exceeds the existing task-oriented dialogue corpora in scale, while also highlighting the challenges associated with building large-scale virtual assistants. It provides a challenging testbed for a number of tasks including language understanding, slot filling, dialogue state tracking and response generation. Along the same lines, we present a schema-guided paradigm for task-oriented dialogue, in which predictions are made over a dynamic set of intents and slots, provided as input, using their natural language descriptions. This allows a single dialogue system to easily support a large number of services and facilitates simple integration of new services without requiring additional training data. Building upon the proposed paradigm, we release a model for dialogue state tracking capable of zero-shot generalization to new APIs, while remaining competitive in the regular setting.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “Dialog.”
Structure-to-Text Generation with Self-Training, Acceptability Classifiers and Context-Conditioning for the GEM Shared Task (2021, methodology): “In this paper, we apply these techniques to 3 of the datasets from the GEM Shared Task (Gehrmann et al., 2021): the Schema-Guided Dialog (SGD) dataset (Rastogi et al., 2019), the End-to-End (E2E) dataset (Novikova et al., 2017) and the WebNLG-en dataset (Gardent et al., 2017).”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, neutral): “…et al., 2019), DART (Radev et al., 2020), ToTTo (Parikh et al., 2020), Wiki-Auto (Jiang et al., 2020), TurkCorpus(Xu et al., 2016) – DialoGLUE: Banking77 (Casanueva et al., 2020), HWU64 (Liu et al., 2019b), CLINC150 (Larson et al., 2019), SGD (Rastogi et al., 2020), TOP (Gupta et al., 2018).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We arbitrarily select XSUM (summarization), ToTTo (table-to-text generation) (Parikh et al., 2020) and Schema Guided Dialog (SGD) (Rastogi et al., 2019) from the GEM benchmark.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “We additionally add a number of task-oriented dialogue tasks: GoogleSGD (Rastogi et al., 2020) and Taskmaster 1, 2 & 3 (Byrne et al., 2019).”
Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “…assume that the agent has access to an external dictionary providing a natural language description descrf of each of these, e.g. “area of the hotel” or “number of hotel database results”, which is common in dialogue state tracking (Rastogi et al., 2020; van Niekerk et al., 2021; Lee et al., 2021).”
Continual Prompt Tuning for Dialog State Tracking (2022, methodology): “ing (DST), a core component of a dialog system, using the Schema-Guided Dialog dataset (Rastogi et al., 2020).”
Continual Learning in Task-Oriented Dialogue Systems (2020, methodology): “In datasets such as SGD, the slot has binary values e.g., yes or no, and thus we exclude those from the count as in Kale and Rastogi (2020).”
Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022, positive): “In our experiments, we have taken K =0 and K =1 to create the dataset for intent detection from SGD and MWOZ datasets.”
Simulated Chats for Task-oriented Dialog: Learning to Generate Conversations from Instructions (2020, methodology): “Crowd-sourcing is a popular method for generating such data-sets and depending on the aspect of dialog modeling being studied, crowd-sourced workers may be asked to annotate existing chat logs for intents and dialog acts (Yu and Yu 2019; Rastogi et al. 2020), create dialog summaries (Gliwa et al.”
MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems (2020, background): “, 2020), which is not scalable when interfacing to a large number of services and APIs spanning multiple domains (Rastogi et al., 2019).”
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2020, methodology): “This is contrasted with modular architectures where the user’s intent is derived from a structured representation and then used to determine which API calls to make such as in Rastogi et al. (2020) and Madotto (2020).”
Few-Shot Natural Language Generation by Rewriting Templates (2020, methodology): “For example, inform, req more and request are some of the dialogue acts defined in the SGD-NLG dataset (Rastogi et al., 2019), which are used for informing the value of a slot to the user, asking if the user needs some other help, and requesting the value of a slot from the user respectively.”
Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue StateTracking (2021, methodology): “These virtual assistants provide natural language interfaces to services and online APIs (Rastogi et al., 2020).”
ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems (2020, background): “…are gaining increasing attention in recent years, resulting in a number of datasets (Henderson et al., 2014; Wen et al., 2017; Budzianowski et al., 2018b; Rastogi et al., 2019) and a wide variety of models (Wen et al., 2015; Peng et al., 2017; Lei et al., 2018; Wu et al., 2019; Gao et al., 2019).”
Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management (2020, methodology): “The idea of using graphs in dialogue systems is not new (Larsen and Baekgaard, 1994; Schlungbaum and Elwert, 1996; Agarwal, 1997), but it is limited to graphs representing flow charts where each node is an action step in a sequence, without a specific semantic importance assigned to the nodes…”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “However, we believe that it has a huge potential in addressing other tasks which require sequential interactions/decisions, such as language and visual dialog [5, 8, 21, 25, 45], dialog navigation [9, 40, 41] and action anticipation for reactive robot response [1, 13, 20].”
CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systems (2021, background): “FewShotSGD Kale and Rastogi (2020a) is the version of the schema-guided-dataset (Rastogi et al., 2019) for natural language generation.”