Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning

By Pararth Shah, Dilek Z. Hakkani-Tür, Bing Liu, Gökhan Tür. 2018

151 citations recorded by the source corpus.

End-to-end neural models show great promise towards building conversational agents that are trained from data and on-line experience using supervised and reinforcement learning. However, these models require a large corpus of dialogues to learn effectively. For goal-oriented dialogues, such datasets are expensive to collect and annotate, since each task involves a separate schema and database of entities. Further, the Wizard-of-Oz approach commonly used for dialogue collection does not provide sufficient coverage of salient dialogue flows, which is critical for guaranteeing an acceptable task completion rate in consumer-facing conversational agents. In this paper, we study a recently proposed approach for building an agent for arbitrary tasks by combining dialogue self-play and crowd-sourcing to generate fully-annotated dialogues with diverse and natural utterances. We discuss the advantages of this approach for industry applications of conversational agents, wherein an agent can be rapidly bootstrapped to deploy in front of users and further optimized via interactive learning from actual users of the system.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “This mirrors other successes in using selfplay, self-chat, and simulated users to evaluate dialogue systems (Fazel-Zarandi et al., 2017; Shah et al., 2018a,b; Wei et al., 2018; Ghandeharioun et al., 2019).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Some of these approaches incorporate human input through explicit, manual feedback (Shah et al., 2018) or implicit signals (e.”
Neural Approaches to Conversational AI (2018, neutral): “ated users. In practice, it is reasonable to have a hybrid approach to evaluation. One possibility is to start with simulated users, then validate or ﬁne-tune the dialogue policy on human users (cf., Shah et al. (2018)). Furthermore, there are more systematic approaches to using both sources of user”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, methodology): “[20] use a task schema and user simulator to generate samples for input to a goal-directed dialog system, while Li et al.”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, methodology): “We evaluate the proposed method on DSTC2 (Henderson et al., 2014a) The movie booking dialogue corpus is generated (Shah et al., 2018) using a ﬁnite state machine based dialogue agent and an agenda based user simulator (Schatzmann et al., 2007) with nat-ural language utterances rewritten by real…”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Most traditional conversational datasets [6, 10, 31, 71] are solely text based, and notoriously difficult to collect, relying on narrowly constructed ontologies [31, 55] and highly specific domains [6, 10].”
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020, background): “Moving towards integrating our ﬁndings into more realistic applications of self-play, e.g., user simulation in dialogue (Schatzmann et al., 2006; Shah et al., 2008), these shortcomings need to be addressed.”
Memory Grounded Conversational Reasoning (2019, methodology): “Wizard-of-Oz Setup : We collect the Memory Di-alog dataset in a Wizard-of-Oz setting (Shah et al., 2018) by connecting two crowd-workers to engage in a role-playing chat session either as a user or an assistant, with the joint goal of creating natural and engaging dialogs (Figure 3).”
Attentive Contextual Carryover for Multi-Turn End-to-End Spoken Language Understanding (2021, methodology): “[34] contains two datasets with a text-only format from Restaurant (11,234 turns in 1,116 training dialogues) and Movie (3,562 turns in 384 training dialogues) domains.”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “…most existing studies use data-driven approaches to build a dialog system from conversational corpora (Zhao and Eskenazi, 2016; Dhingra et al., 2017; Shah et al., 2018; Shi and Yu, 2018), where a common strategy is to build a user simulator, and then to learn dialog policy through making…”
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020, methodology): “• GSIM (Shah et al., 2018a): GSIM is a humanrewrote machine-machine task-oriented corpus, including 1500/469/1039 dialogues for the train,”
ChartDialogs: Plotting from Natural Language Instructions (2020, unknown): “In particular, we compare to: DSTC2 (Henderson et al., 2014), SFX (Gaˇsi´c et al., 2014), WOZ (Wen et al., 2017), FRAMES (Asri et al., 2017), KVRET (Eric and Manning, 2017), M2M (Shah et al., 2018) and Im-ageEdits (Manuvinakurike et al., 2018b,a).”
OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs (2019, methodology): “Wizard-of-Oz setup The dialogs were generated in a Wizard-of-Oz setting (Shah et al., 2018) by connecting two crowd-workers to engage in a chat session, with the joint goal of creating natural and engaging dialogs.”
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems (2020, background): “…learn a better dialogue policy via interaction (Asri et al., 2016; Li et al., 2017; Wu et al., 2019c; Peng et al., 2018), and it is especially useful in scenarios in where few or no data is available (Liu and Lane, 2017; Liu et al., 2017; Shah et al., 2018; Kreyssig et al., 2018; Li et al., 2020).”
A Repository of Conversational Datasets (2019, background): “, 2017) comprises 19,986 turns, and M2M (Shah et al., 2018) spans 14,796 turns.”