Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System

By Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, Susannah Young. North American Chapter of the Association for Computational Linguistics, 2007

423 citations recorded by the source corpus.

This paper investigates the problem of bootstrapping a statistical dialogue manager without access to training data and proposes a new probabilistic agenda-based method for simulating user behaviour. In experiments with a statistical POMDP dialogue system, the simulator was realistic enough to successfully test the prototype system and train a dialogue policy. An extensive study with human subjects showed that the learned policy was highly competitive, with task completion rates above 90%.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f78b22cda8df1b59c54e2df04a966085aa865835

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Task-oriented Dialogue System for Automatic Diagnosis (2018, background): “In practice, the user state s u is factored into an agenda A (Schatzmann et al., 2007) and a goal G , noted as s u = ( A, G ) .”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “…with humans, and offline using user simulator models and reinforcement learning methods (Levin et al., 1997; Georgila et al., 2006; Paek, 2006; Schatzmann et al., 2007; Jung et al., 2009; Schatzmann and Young, 2009; Gašić et al., 2010, 2011; Daubigney et al., 2012; Gašić et al., 2012; Su…”
Phrase-Based Statistical Language Generation Using Graphical Models and Active Learning (2010, background): “…presentation dialogue acts produced over 20,000 simulated dialogues between our statistical dialogue manager (Young et al., 2010) and an agenda-based user simulator (Schatzmann et al., 2007), which results in 202 unique dialogue acts after replacing nonenumerable values by a generic symbol.”
Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding (2018, methodology): “(Eckert et al., ; Schatzmann et al., 2007a; Asri et al., 2016; Scheffler and Young, 2000; Scheffler and Young, 2001; Pietquin and Dutoit, 2006; Georgila et al., 2005; Cuayáhuitl et al., 2005).”
Sample Efficient Deep Reinforcement Learning for Dialogue Systems With Large Action Spaces (2018, background): “The agendabased user simulator [39] consists of a goal which is a randomly generated slot-value pairs that the entity that the user seeks must be satisfied and an agenda which is a dynamic stack of dialogue acts that the user elicits in order to satisfy the goal.”
Policy committee for adaptation in multi-domain spoken dialogue systems (2015, background): “In order to investigate the effectiveness of the proposed policy committee model, a variety of contrasts were examined using an agenda-based simulated user operating at the dialogue act level [19, 20].”
The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach (2018, methodology): “…using abstract dialogue states and actions (Eckert et al., 1997; Levin et al., 2000; Chung, 2004; Cuay´ahuitl et al., 2005; Georgila et al., 2006; Schatzmann et al., 2007; Heeman, 2009; Traum et al., 2008; Lee & Eskenazi, 2012; Khouzaimi et al., 2017; L´opez-C´ozar, 2016; Su et al., 2016;…”
A Deep Reinforcement Learning Chatbot (2017, background): “s focused on learning dialogue systems through simulations using abstract dialogue states and actions (Eckert et al. 1997, Levin et al. 2000, Chung 2004, Cuayáhuitl et al. 2005, Georgila et al. 2006, Schatzmann et al. 2007, Heeman 2009, Traum et al. 2008, Georgila & Traum 2011, Lee & Eskenaz”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, methodology): “In the task-oriented dialog setting, the entire conversation is around a user goal G = ( C, R ) implicitly, where C denotes the constraint and R is the requests (Schatzmann et al., 2007).”
Data-Driven Methods for Adaptive Spoken Dialogue Systems (2012, methodology): “Although the agenda-based user simulator introduced by [40] was entirely handcrafted, it was realistic enough to successfully test a prototype POMDP dialogue manager and train a dialogue policy that outperformed a handcrafted baseline [45].”
Collaborative Multi-Agent Dialogue Model Training Via Reinforcement Learning (2019, methodology): “Employing a user simulator is an established method for dialogue policy learning (Schatzmann et al., 2007, among others) and end-to-end dialogue training (Asri et al., 2016; Liu and Lane, 2018b).”
Distributed dialogue policies for multi-domain statistical dialogue management (2015, background): “Training and testing was performed on an agendabased simulated user operating on the dialogue act level [17, 18].”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, methodology): “In Section 2.2, we extend the user goal deﬁnition from Schatzmann et al. (2007) to cover a broad concept that we deﬁne as user goal query .”
Building a Conversational Agent Overnight with Dialogue Self-Play (2018, methodology): “In our setup, B U is an agenda-based user simulator (Schatzmann et al. (2007)) with a modiﬁ-cation that the action selection model is conditioned on the user proﬁle in addition to the user goal and dialogue history.”
Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval (2015, methodology): “the fitted value iteration (FVI) [357] algorithm) using a corpus of historical data of user interactions, or simulated users generated based on some of such data [358].”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, methodology): “The coupling ofﬂine of agent training with simulation of user behavior was studied by Schatzmann et al. [2007] using a rule-based approach.”
Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems (2021, methodology): “However, expert knowledge is required to design their rules and the number of rules needed for complex domains quickly becomes intractable (Schatzmann et al., 2007).”
k-Nearest Neighbor Monte-Carlo Control Algorithm for POMDP-Based Dialogue Systems (2009, background): “The optimal policy is obtained using reinforcement learning in interaction with an agenda based simulated user (Schatzmann et al., 2007).”