The Second Dialog State Tracking Challenge

By Matthew Henderson, Blaise Thomson, Jason Williams. 2014

607 citations recorded by the source corpus.

A spoken dialog system, while communicating with a user, must keep track of what the user wants from the system at each step. This process, termed dialog state tracking, is essential for a successful dialog system as it directly informs the system’s actions. The first Dialog State Tracking Challenge allowed for evaluation of different dialog state tracking techniques, providing common testbeds and evaluation suites. This paper presents a second challenge, which continues this tradition and introduces some additional features ‐ a new domain, changing user goals and a richer dialog state. The challenge received 31 entries from 9 research groups. The results suggest that while large improvements on a competitive baseline are possible, trackers are still prone to degradation in mismatched conditions. An investigation into ensemble learning demonstrates the most accurate tracking can be achieved by combining multiple trackers.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Traditional goal-oriented dialogue has focused on narrow tasks that would typically be useful for a dialogue-based assistant, for example restaurant (Henderson, Thomson, and Williams, 2014), taxi, train, and hotel (Budzianowski et al., 2018) or trip (El Asri et al., 2017) booking.”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, background): “Goal-oriented tasks typically have a clearer evaluation, e.g. task completion can be measured if the correct actions are taken (Hastie 2012; Henderson, Thomson, and Williams 2014; Bordes, Boureau, and Weston 2017; El Asri et al. 2017; Wen et al. 2017).”
Wizard of Wikipedia: Knowledge-Powered Conversational agents (2018, background): “ing it directly into the weights of the feed-forward network. In the area of goal-directed dialogue, separate from open domain chit-chat, such as airline (El Asri et al., 2017) or restaurant booking (Henderson et al., 2014; Wen et al., 2016; Bordes et al., 2017), knowledge conditioning is typically ”
Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “Some of these datasets contain transcripts of human-bot conversations (Williams et al., 2013; Henderson et al., 2014a,b) while others are created using a ﬁxed set of natural language patterns (Bordes and Weston, 2017; Dodge et al., 2016).”
Neural Approaches to Conversational AI (2018, neutral): “ures for dialogue state tracking. Starting from Williams et al. (2013), it has successfully attracted many research teams to focus on a wide range of technical problems in DST (Williams et al., 2014; Henderson et al., 2014b,a; Kim et al., 2016a,b; Hori et al., 2017). Corpora used by DSTC over the ye”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “Dialogue Dialogue evaluation is relatively well understood in goal-oriented tasks, where automated approaches can be coded by measuring task completion (Bordes et al., 2017; El Asri et al., 2017; Hastie, 2012; Henderson et al., 2014; Wen et al., 2017).”
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems (2015, background): “Switchboard [7] Human-human Various 2,400 — 3,000,000 Telephone conversations spoken on pre-specified topics DSTC1 [30] Human-computer State 15,000 210,000 Bus ride information spoken tracking system DSTC2 [9] Human-computer State 3,000 24,000 — Restaurant booking spoken tracking system DSTC3 [8] Hu”
Learning End-to-End Goal-Oriented Dialog (2016, methodology): “However, in spite of recent efforts to learn modules (Henderson et al. , 2014b), they still require many hand-crafted features for the state and action space representations, which restrict their usage to narrow domains.”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “Ultimately the impact of any unnaturalness in the dialogues depends on the task and context in which the dialogue system is deployed.”
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems (2015, methodology): “…of the dialog state tracking stage is well deﬁned since the PAR-ADISE framework (Walker et al., 1997) and subsequent initiatives (Paek, 2001; Griol et al., 2008), including recent competitons (Williams et al., 2013; Henderson et al., 2014) as well as situated variants (Rojas-Barahona et al., 2012).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “We focus on restaurant recommendations in our experiments, which, previously, have been widely explored in dialogue systems research, e.g. (Young et al. 2010; Henderson, Thomson, and Williams 2014; Wen et al. 2017).”
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, methodology): “To illustrate the contribution of the new corpus, we compare it on several important statistics with the DSTC2 corpus (Henderson et al., 2014a), the SFX corpus (Gašić et al.”
Continuously Learning Neural Dialogue Management (2016, methodology): “As depicted in Figure 1, the input to the model is the belief state s which encodes the understood user in-tents along with the dialogue history (Henderson et al., 2014b; Mrkˇsi´c et al., 2015), and the output is the master dialogue action a that decides the semantic reply.”
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset (2019, background): “The Dialogue State Tracking Challenges (Williams et al. 2013; Henderson, Thomson, and Williams 2014a; 2014b; Kim et al. 2017) contributed to the creation of dialogue datasets with increasing complexity.”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, background): “…are often at the end of the dialogue, and partners rarely reply with further negotiation—so the models using rollouts and reinforcement learning believe this tactic will help their offer to be accepted. tated in the training data (Williams and Young, 2007; Henderson et al., 2014; Wen et al., 2016).”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “Most recent work in dialogue exploring generative or retrieval models for goal-directed (Henderson et al., 2014; Bordes et al., 2017) or chitchat tasks (Vinyals and Le, 2015; Sordoni et al.”
Word-Based Dialog State Tracking with Recurrent Neural Networks (2014, methodology): “This section describes the domain and methodology of the second Dialog State Tracking Challenge (DSTC2).”
TextWorld: A Learning Environment for Text-based Games (2018, background): “This enables exact state-tracking [Henderson et al., 2014] and the corresponding assignment of intermediate rewards in training (if desired).”