A Deep Reinforcement Learning Chatbot

By Iulian Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, A. P. Sarath Chandar, Nan Rosemary Ke and 8 more. 2017

245 citations recorded by the source corpus.

We present MILABOT: a deep reinforcement learning chatbot developed by the Montreal Institute for Learning Algorithms (MILA) for the Amazon Alexa Prize competition. MILABOT is capable of conversing with humans on popular small talk topics through both speech and text. The system consists of an ensemble of natural language generation and retrieval models, including template-based models, bag-of-words models, sequence-to-sequence neural network and latent variable neural network models. By applying reinforcement learning to crowdsourced data and real-world user interactions, the system has been trained to select an appropriate response from the models in its ensemble. The system has been evaluated through A/B testing with real-world users, where it performed significantly better than many competing systems. Due to its machine learning architecture, the system is likely to improve with additional data.

Subfields: Artificial Intelligence, Computer Science Applications, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, background): “Some open-domain chatbots such as MILABOT (Serban et al., 2017), XiaoIce (Zhou et al., 2018)1, Gunrock (Chen et al., 2018), Mitsuku (Worswick, 2018)2 and Cleverbot3 (by Rollo Carpenter) display human-like attributes, but rely on com- 1https://www.msxiaobing.com/…”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Some attempts at predicting a quality score from a range of different signals have shown some positive results (Serban et al., 2017; Fang et al., 2018; Ghandeharioun et al., 2019), and have been used to train models through reinforcement learning (Serban et al.”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, methodology): “In particular (Serban et al., 2017) employ user sentiment detection for reward shaping in their Alexa prize entry.”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “Even in this setting, authors may choose to use a highly restricted action space; for example, using RL to choose which scripted or MLE dialog model to invoke to answer a user’s query [51].”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “There has been recent work which combines responses from multiple sub-systems (Serban et al., 2017; Papaioannou et al., 2017) and ranks them to output the ﬁnal sys-tem response.”
Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “We train two commonly used end-to-end dialogue models, HRED (Serban et al. 2016) and VHRED (Serban et al. 2017c), on the Twitter dataset and sample responses using contexts from the validation set.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “The underlying parameters of the VHRED model were as follows: Context RNN hidden size = 1000 , decoder hidden size = 1250 , encoder hidden size = 1250 , z embedding size = 600 , gradient clip = 1 .”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, methodology): “For example, modern solutions that build an open-ended dialogue system to the Alexa challenge combine hand-coded and machine-learned elements (Serban et al., 2017a).”
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents (2019, background): “…in the capabilities of these neural network architectures and developments (Serban et al., 2016; Miao, Yu, and Blunsom, 2015; Sordoni et al., 2015; Serban et al., 2017; Li, Monroe, and Jurafsky, 2016; Li et al., 2017) indicated that they were limited which made communicating with them a rather…”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “…dialogue system architectures based on neural networks have shown promising results on several dialogue tasks (Ritter et al., 2011; Vinyals and Le, 2015; Lowe et al., 2015a; Sordoni et al., 2015b; Shang et al., 2015; Li et al., 2015; Serban et al., 2016; Serban et al., 2017b,a; Dodge et al., 2015).”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “a representation is found it allows for interpretable latent code manipulation, which is desirable in a variety of applications; recently, for example, in reinforcement learning [39, 18, 11, 41, 34].”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, background): “the Alexa prize [15, 16], or the Conversational Intelligence Challenge [6]).”
Deploying Lifelong Open-Domain Dialogue Learning (2020, background): “…where the feedback is a scalar rather than the dialogue messages themselves (Levin et al., 2000; Schatzmann et al., 2006; Rieser and Lemon, 2011; Liu and Lane, 2017; Serban et al., 2017), which is most common in a goal-oriented setting where completion of the goal can provide such rewards.”
Assessing The Factual Accuracy of Generated Text (2019, background): “Recently, there has been wide empirical success in text summarization [15, 21, 27], machine translation [1, 36, 39], dialogue response generation [12, 28, 29], and other text generation tasks.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, methodology): “[60] integrated natural language generation and retrieval models, including template-based models, bag-of-words models, sequence-to-sequence neural network and latent variable neural network models and applied reinforcement learning to crowdsourced data and real-world user interactions to select”
Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning (2022, methodology): “[32] proposed MILABOT, as part of the Amazon Alexa Prize competition, where a DM selects a response from several generated candidates.”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “The most similar work to ours is Nogueira dos Santos et al. (2018) who trained for the ﬁrst time an encoder-decoder rewriting offensive sentences in a non-offensive register with non-parallel data from Twitter (Ritter et al., 2010) and Reddit (Serban et al., 2017).”