The Dialog State Tracking Challenge

By Jason Williams, Antoine Raux, Deepak Ramachandran, Alan W. Black. 2013

380 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “Some of these datasets contain transcripts of human-bot conversations (Williams et al., 2013; Henderson et al., 2014a,b) while others are created using a ﬁxed set of natural language patterns (Bordes and Weston, 2017; Dodge et al., 2016).”
Neural Approaches to Conversational AI (2018, neutral): “ipeline approaches (c.f., Sec. 4.6). Dialogue State Tracking Challenge (DSTC) is a series of challenges that provide common testbeds and evaluation measures for dialogue state tracking. Starting from Williams et al. (2013), it has successfully attracted many research teams to focus on a wide range o”
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems (2015, methodology): “The dataset is orders of magnitude larger than structured corpuses such as those of the Dialogue State Tracking Challenge [32].”
A Deep Reinforcement Learning Chatbot (2017, background): “The idea of modeling a high-level abstraction of the dialogue, z t , is related to the dialogue state tracking challenge (Williams et al. 2013, 2016).”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “, 2000), dialogue state tracking (Thomson and Young, 2010; Wang and Lemon, 2013; Ren et al., 2013; Henderson et al., 2013; Williams et al., 2013; Henderson et al., 2014c; Kim et al., 2015), natural language generation (Langkilde and Knight, 1998; Oh and Rudnicky, 2000; Walker et al.”
The Second Dialog State Tracking Challenge (2014, methodology): “One rule uses a learnt parameter called the noise adjustment, to adjust the SLU scores.”
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (2019, background): “Speciﬁcally, DSTC 1 (Williams et al., 2013), DSTC 2 (Hen-derson et al., 2014a), and DSTC 3 (Henderson et al., 2014b) contain “chatbot style” queries, but the datasets are focused on state tracking.”
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, methodology): “…to label the semantics, these corpora can be roughly divided into two categories: corpora with structured semantic labels (Hemphill et al., 1990; Williams et al., 2013; Asri et al., 2017; Wen et al., 2017; Eric et al., 2017; Shah et al., 2018); and corpora without semantic labels but with an…”
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset (2019, background): “The Dialogue State Tracking Challenges (Williams et al. 2013; Henderson, Thomson, and Williams 2014a; 2014b; Kim et al. 2017) contributed to the creation of dialogue datasets with increasing complexity.”
Multi-domain Dialog State Tracking using Recurrent Neural Networks (2015, background): “The recent Dialog State Tracking Challenge (DSTC) shared tasks (Williams et al., 2013; Henderson et al., 2014a; Henderson et al., 2014b) saw a variety of novel approaches, including robust sets of hand-crafted rules (Wang and Lemon, 2013), conditional random fields (Lee and Eskenazi, 2013; Lee, 2013”
Reference-Aware Language Models (2016, methodology): “We use dialogues from the second Dialogue State Tracking Challenge (DSTC2) (Henderson et al., 2014).”
Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “Another well-known corpus is derived from a series of competitions on the task of dialogue-state tracking (Williams et al., 2013).”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “The system action is defined with the act and slot types from a dialogue act (Henderson et al., 2013).”
Neural Belief Tracker: Data-Driven Dialogue State Tracking (2016, background): “As mentioned above, the DSTC shared tasks have spurred research on this problem and institutionalised a standard evaluation paradigm (Williams et al., 2013; Henderson et al., 2014b; Henderson et al., 2014a).”
Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems (2016, background): “In the above scenarios, the dialogs are domain-speciﬁc, and a typical approach to such in-domain systems is by human engineering, for example, using manually constructed ontologies [23], natural language templates [20], and even predeﬁned dialog states [25].”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “8]. A statistical dialog system maintains a distribution over multiple hypotheses of the true dialog state, facing with noisy conditions and ambiguity [104]. In Dialog State Tracking Challenge (DSTC) [87; 86], the results are in the form of a probability distribution over each slot for each turn. A ”
Learning to Control the Specificity in Neural Response Generation (2018, background): “At the very beginning, people started the research using handcrafted rules and templates (Walker et al., 2001; Williams et al., 2013; Henderson et al., 2014).”
Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning (2016, methodology): “Most industrial systems use rule-based heuristics to update the dialog state by selecting a high-conﬁdence output from the NLU (Williams et al., 2013).”
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (2018, background): “Conversational agents primarily fall into two categories: task oriented dialogue systems (Williams et al., 2013; Wen et al., 2015) and chatbots (Weizenbaum, 1966), although there have been some efforts to integrate the two (Dodge et al., 2015; Yu et al., 2017).”