A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies

By Jost Schatzmann, Karl Weilhammer, Matthew N. Stuttle, Steve J. Young. 2006

Subfields: Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “…et al., 2018; Hixon et al., 2015; Pappu and Rudnicky, 2013), and more commonly in reinforcement learning settings, where the feedback is a scalar rather than the dialogue messages themselves (Levin et al., 2000; Schatzmann et al., 2006; Rieser and Lemon, 2011; Liu et al., 2018; Hong et al., 2019).”
Dialog-based Language Learning (2016, neutral): “C L ] 1 8 M ay 2 01 6”
Deep Reinforcement Learning for Dialogue Generation (2016, background): “This dialogue literature thus widely applies reinforcement learning (Walker, 2000; Schatzmann et al., 2006; Gasic et al., 2013b; Singh et al., 1999; Singh et al., 2000; Singh et al., 2002) to train dialogue policies.”
Neural Approaches to Conversational AI (2018, neutral): “non-static user goals during the conversations, among others. Here, we highlight two dimensions, and refer interested users to a survey for further details on creating and evaluating user simulators (Schatzmann et al., 2006): Along the granularity dimension, the user simulator can operate either at ”
Neural Responding Machine for Short-Text Conversation (2015, background): “Previous works in this direction mainly focus on either rule-based or learning-based methods (Williams and Young, 2007; Schatzmann et al., 2006; Misu et al., 2012; Litman et al., 2000).”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “User simulation models are particularly useful in the development of dialogue systems based on reinforcement learning methods (Singh et al., 2002; Schatzmann et al., 2006; Pietquin and Dutoit, 2006; Frampton and Lemon, 2009; Jurčı́ček et al., 2012; Png and Pineau, 2011; Young et al., 2013).”
Dialogue Learning With Human-In-The-Loop (2016, neutral): “Reinforcement learning has been widely applied to dialogue, especially in slot ﬁlling to solve domain-speciﬁc tasks (Walker, 2000; Schatzmann et al., 2006; Singh et al., 2000; 2002).”
Continuously Learning Neural Dialogue Management (2016, methodology): “The policy network was tested with a simulated user (Schatzmann et al., 2006) which provides the interaction at the semantic level.”
Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets (2008, background): “…amount of dialogue system work that has relied on simulated users (e.g., Schefﬂer and Young 2002; Pietquin 2004; Georgila, Henderson, and Lemon 2006; Schatzmann et al. 2006), to date there has not been a systematic experiment that validates this methodology against results from human users.”
Frames: a corpus for adding memory to goal-oriented dialogue systems (2017, background): “As for dialogue management, simulators have been proposed (Schatzmann et al., 2006) but datasets are scarce.”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, methodology): “The proposed models were ﬁrst evaluated under 0% semantic error rate with an agenda-based simulator which generates user interactions at the semantic-level (Schatzmann et al., 2006).”
Deploying Lifelong Open-Domain Dialogue Learning (2020, background): “…is common in reinforcement learning settings, where the feedback is a scalar rather than the dialogue messages themselves (Levin et al., 2000; Schatzmann et al., 2006; Rieser and Lemon, 2011; Liu and Lane, 2017; Serban et al., 2017), which is most common in a goal-oriented setting where…”
Iterative policy learning in end-to-end trainable task-oriented neural dialog models (2017, background): “Therefore, a user simulator [21, 22, 23] is usually used to train the dialog agent to a sufficient starting level before deploying it to the real environment.”
Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols (2017, background): “Usually, they are represented by hybrid models that combine reinforcement learning with supervised learning (Henderson et al., 2008; Schatzmann et al., 2006).”
Improving Mild Cognitive Impairment Prediction via Reinforcement Learning and Dialogue Simulation (2018, methodology): “Such dialogue systems are typically designed to be a pipeline containing a set of components including a language understanding unit that parses the intention and semantics from the input from humans, a dialogue manager that manages dialogue state tracking and policy learning, and a language generat”
Task-oriented Dialogue System for Automatic Diagnosis (2018, background): “Typically, a user simulator is designed to interact with the dialogue system (Liu et al., 2017; Peng et al., 2017b; Su et al., 2016; Schatzmann et al., 2006).”
Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System (2007, methodology): “A statistical user model is first trained on a limited amount of dialogue data and the model is then used to simulate dialogues with the interactively learning DM (see Schatzmann et al. (2006) for a literature review).”