Building a Conversational Agent Overnight with Dialogue Self-Play

By Pararth Shah, Dilek Z. Hakkani-Tür, Gökhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak Kennard, Larry Heck. 2018

208 citations recorded by the source corpus.

We propose Machines Talking To Machines (M2M), a framework combining automation and crowdsourcing to rapidly bootstrap end-to-end dialogue agents for goal-oriented dialogues in arbitrary domains. M2M scales to new tasks with just a task schema and an API client from the dialogue system developer, but it is also customizable to cater to task-specific interactions. Compared to the Wizard-of-Oz approach for data collection, M2M achieves greater diversity and coverage of salient dialogue flows while maintaining the naturalness of individual utterances. In the first phase, a simulated user bot and a domain-agnostic system bot converse to exhaustively generate dialogue "outlines", i.e. sequences of template utterances and their semantic parses. In the second phase, crowd workers provide contextual rewrites of the dialogues to make the utterances more natural while preserving their meaning. The entire process can finish within a few hours. We propose a new corpus of 3,000 dialogues spanning 2 domains collected with M2M, and present comparisons with popular dialogue datasets on the quality and diversity of the surface forms and dialogue flows.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Bing Liu

In Natural Language Processing

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “This mirrors other successes in using selfplay, self-chat, and simulated users to evaluate dialogue systems (Fazel-Zarandi et al., 2017; Shah et al., 2018a,b; Wei et al., 2018; Ghandeharioun et al., 2019).”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “This mirrors other successes in using self-play, self-chat, and simulated users to evaluate dialogue systems (Fazel-Zarandi et al., 2017; Shah et al., 2018a,b; Wei et al., 2018; Ghandeharioun et al., 2019).”
MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling (2018, methodology): “…two categories: corpora with structured semantic labels (Hemphill et al., 1990; Williams et al., 2013; Asri et al., 2017; Wen et al., 2017; Eric et al., 2017; Shah et al., 2018); and corpora without semantic labels but with an implicit user goal in mind (Ritter et al., 2010; Lowe et al., 2015).”
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset (2019, methodology): “0 (Wen et al. 2017), FRAMES (El Asri et al. 2017), M2M (Shah et al. 2018) and MultiWOZ (Budzianowski et al. 2018).”
Bootstrapping a Neural Conversational Agent with Dialogue Self-Play, Crowdsourcing and On-Line Reinforcement Learning (2018, background): “We direct the reader to the technical report Shah et al. (2018) for a detailed description of this approach.”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Task-oriented Dialogue Shah et al. (2018) SGD A schema-guided dataset spanningover multiple domains.”
Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management (2020, methodology): “We additionally append an artificial final state to each dialogue to explicitly mark the end of the conversation for datasets where a "task_complete" indicator is not present (Shah et al., 2018).”
Learning from On-Line User Feedback in Neural Question Answering on the Web (2019, background): “Applications in these disciplines include user feedback that is leveraged in order to improve conversational agents [36], assess the relevance of answers in search queries for knowledge graphs [39], or identify incorrect queries to databases [18].”
Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking (2020, methodology): “Various semantic parsing datasets have been generated with this technique (Su et al., 2017; Zhong et al., 2017) and the technique has also been adapted to the multiturn setting (Cheng et al., 2018; Shah et al., 2018).”
MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines (2020, background): “Many public datasets like DSTC2 (Hen-derson et al., 2014), WOZ (Wen et al., 2017a), SimulatedDialogue (Shah et al., 2018), Multi-WOZ (Budzianowski et al., 2018), TaskMas-ter (Byrne et al., 2019), SGD (Rastogi et al., 2019), etc. have been very useful to facilitate research in this area.”
Dialogue State Tracking with a Language Model using Schema-Driven Prompting (2021, background): “Machines Talking To Machines (M2M) (Shah et al., 2018) is a framework that combines simulation and online crowdsourcing.”
Multi-task Learning for Joint Language Understanding and Dialogue State Tracking (2018, background): “The dataset, described in Shah et al. 2017, contains dialogues from restaurant (Sim-R) and movie (Sim-M) domains across three intents.”
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling (2021, methodology): “After the dialogue paraphrasing, workers are asked to read through the new dialogue and answer the following questions, as in [23]: 1) Does it seem like a conversation between a user that sounds like you and an assistant that sounds formal? 2) Does it have the same meaning as the original conversati”
TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking (2020, methodology): “We train and test our model on four datasets, Mul-tiWOZ 2.1 (Eric et al., 2019), WOZ 2.0 (Wen et al., 2016), sim-M and sim-R (Shah et al., 2018).”