The Dialog State Tracking Challenge Series: A Review

By Jason Williams, Antoine Raux, Matthew Henderson. 2016

241 citations recorded by the source corpus.

In a spoken dialog system, dialog state tracking refers to the task of correctly inferring the state of the conversation – such as the user’s goal – given all of the dialog history up to that turn. Dialog state tracking is crucial to the success of a dialog system, yet until recently there were no common resources, hampering progress. The Dialog State Tracking Challenge series of 3 tasks introduced the first shared testbed and evaluation metrics for dialog state tracking, and has underpinned three key advances in dialog state tracking: the move from generative to discriminative models; the adoption of discriminative sequential techniques; and the incorporation of the speech recognition results directly into the dialog state tracker. This paper reviews this research area, covering both the challenge tasks themselves and summarizing the work they have enabled.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “While there are already some works in this direction (Rojas-Barahona et al., 2017; Williams et al., 2016; Lowe et al., 2015a; Ghazvininejad et al., 2017) which try to integrate external knowledge sources with existing datasets, we believe that building new datasets where the utterances are…”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “(Williams et al., 2016) Specifically, it captures the user goals in the conversation in the form of (slot, value) pairs.”
End-to-end LSTM-based dialog control optimized with supervised and reinforcement learning (2016, methodology): “For example, in the Dialog State Tracking Challenge (DSTC), the state consisted of a pre-specified frame of name/value pairs that form the user’s goal (Williams et al., 2016).”
Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize (2018, methodology): “Apart from the Alexa Prize, there have been challenges like Dialog System Technology Challenge (DSTC) (Williams et al., 2016) (task based and closed domain) and Conversational AI Challenge (ConvAI) (Burtsev et al., 2018) (persona based, chit-chat and challenges with evaluation).”
Neural Belief Tracker: Data-Driven Dialogue State Tracking (2016, background): “The Dialogue State Tracking Challenge (DSTC) series of shared tasks has provided a common evaluation framework accompanied by labelled datasets (Williams et al., 2016).”
Frames: a corpus for adding memory to goal-oriented dialogue systems (2017, methodology): “Frame tracking extends state tracking (Henderson, 2015; Williams et al., 2016) to a setting where several semantic frames are tracked simultaneously.”
Iterative policy learning in end-to-end trainable task-oriented neural dialog models (2017, methodology): “User Goal We define a user’s goal g using a list of informable and requestable slots [39].”
Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset (2019, methodology): “Many of the existing task-based datasets have been generated from deployed dialog systems such as the Lets Go Bus Information System (Raux et al., 2003) and the various Dialog State Tracking Challenges (DSTCs) (Williams et al., 2016).”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “A dialog state tracker keeps track of the dialog session to update the dialog state (Williams et al., 2016; Zhang et al., 2019).”
Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialogue State Tracking (2021, background): “Dialog state tracking (DST) extracts users’ goals in task-oriented dialog systems, where dialog states are often represented in terms of a set of slot-value pairs (Williams et al., 2016; Eric et al., 2020).”
A Conversational (methodology): “Our design is based on the Dialogue-State Architecture described by Williams et al. [23].”
Further Advances in Open Domain Dialog Systems in the Third Alexa Prize Socialbot Grand Challenge (2020, methodology): “Apart from the Alexa Prize, there have been challenges like Dialog System Technology Challenge (DSTC) (Williams et al., 2016) (task based and closed domain) and Conversational AI Challenge (ConvAI) (Burtsev et al., 2018) (persona based, chit-chat and challenges with evaluation).”
Dialog state tracking, a machine reading approach using Memory Network (2016, background): “Recently, deep learning based models implementing this strategy (Mrksic et al., 2016; Henderson et al., 2014a; Williams et al., 2016) have shown state of the art results.”
Semantically Conditioned Dialog Response Generation via Hierarchical Disentangled Self-Attention (2019, unknown): “domain datasets like DSTC (Williams et al., 2016), CamRes767 (Rojas-Barahona et al.”
Semantic Parsing for Task Oriented Dialog using Hierarchical Representations (2018, methodology): “Most work on task oriented dialog systems has focused on identifying a single user intent and then ﬁlling the relevant slots – for example, the representations used on the ATIS dataset (Mesnil et al., 2013; Liu and Lane, 2016; Zhu and Yu, 2017) and in the Dialog State Tracking Challenge (Williams et”
Machine Learning for Dialog State Tracking: A Review (2015, background): “Nevertheless, full results for all the DSTCs can be found in the relevant summary papers [1–3] and in the overview paper [4].”
Dialog State Tracking with Reinforced Data Augmentation (2019, background): “DST is studied extensively in the literature (Williams et al., 2016 methods can be classiﬁed into three categories: rule-based (Zue et al., 2000), generative (DeVault and Stone, 2007; Williams, 2008), and discriminative (Metallinou et al., 2013) methods.”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Similarly, for certain types of goaloriented conversations more targeted evaluations can take place, for example evaluation of state tracking (Williams et al., 2016) and task completion (Hastie, 2012; Henderson et al.”
Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings (2017, unknown): “Most work focuses on information-querying tasks, using Wizard-of-Oz data collection (Williams et al., 2016; Asri et al., 2016) or simulators (Bordes and Weston, 2017; Li et al., 2016d), In contrast, collaborative dialogues are easy to collect as natural human conversations, and are also challenging…”