The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems

By Ryan J. Lowe, Nissan Pow, Iulian Serban, Joelle Pineau. 2015

981 citations recorded by the source corpus.

This paper introduces the Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words. This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data. The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter. We also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next response.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “Ubuntu Dialogue Generation (Lowe et al., 2015) 27.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Nevertheless, a number of studies do report automatic metrics, sometimes without human studies (Lowe et al., 2015; Serban et al., 2016; Parthasarathi and Pineau, 2018).”
Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “datasets (Lowe et al., 2015b; Ritter et al., 2010; Sordoni et al., 2015) leading to the development of several end-to-end conversation systems (Shang et al.”
Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring (2019, methodology): “Ubuntu V2 is a similar but larger popular corpus, created before the competition (Lowe et al., 2015); we report results for this dataset as well, as there are many existing results on it.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “Recently, there has been signiﬁcant interest in building chatbots (Sordoni et al., 2015; Wen et al., 2015) fueled by the availability of popular dialog data sets such as Ubuntu, Twitter, and Movie dialogs (Lowe et al., 2015; Rit-ter et al., 2011; Danescu-Niculescu-Mizil and Lee, 2011).”
Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “2017b), the Cornell Movie Dialogue Corpus (Danescu-Niculescu-Mizil and Lee 2011), and the Ubuntu Dialogue Corpus (Lowe et al. 2015).”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, methodology): “We then evaluate the task using three metrics: (i) the log likelihood of the correct sequence, measured via perplexity, (ii) F1 score, and (iii) next utterance classiﬁcation loss, following Lowe et al. (2015).”
ParlAI: A Dialog Research Software Platform (2017, methodology): “Over 20 tasks are supported in the first release, including popular datasets such as SQuAD (Rajpurkar et al., 2016), bAbI tasks (Weston et al., 2015), QACNN and QADailyMail (Hermann et al., 2015), CBT (Hill et al., 2015), bAbI Dialog tasks (Bordes and Weston, 2016), Ubuntu (Lowe et al., 2015) and VQ”
Neural Approaches to Conversational AI (2018, positive): “• Ubuntu: The Ubuntu dataset (Lowe et al., 2015) has also been used extensively for E2E conversation modeling.”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, methodology): “We also experiment on the Ubuntu Dialogue Corpus [17], which contains about 500, 000 dialogues extracted from the #Ubuntu Internet Relayed Chat channel.”
What makes a good conversation? How controllable attributes affect human judgments (2019, background): “Nevertheless, a number of studies only use automated metrics, with no human study at all (Lowe et al., 2015; Serban et al., 2016b; Parthasarathi and Pineau, 2018).”
Learning End-to-End Goal-Oriented Dialog (2016, methodology): “Classical information retrieval (IR) models with no machine learning are standard baselines that often perform surprisingly well on dialog tasks (Isbell et al. , 2000; Jafarpour et al. , 2010; Ritter et al. , 2011; Sordoni et al. , 2015).”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “…4 sources of candidate responses: (1) a response selected by a TF-IDF retrieval-based model, (2) a response selected by the Dual Encoder (DE) (Lowe et al., 2015), (3) a response generated using the hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a), and (4)…”
A Deep Reinforcement Learning Chatbot (2017, methodology): “Dual Encoder Models : The system contains two Dual Encoder retrieval models (Lowe et al. 2015), DualEncoderRedditPolitics and DualEncoderRedditNews .”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “Ultimately the impact of any unnaturalness in the dialogues depends on the task and context in which the dialogue system is deployed.”
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems (2015, methodology): “To validate the quality of our results, we also apply our best performing model, Memory Networks, in other conditions by comparing it on the Ubuntu Dialog Corpus (Lowe et al., 2015) against baselines trained by the authors of the corpus.”