A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version

By Iulian Serban, Ryan J. Lowe, Peter Henderson, Laurent Charlin, Joelle Pineau. 2015

358 citations recorded by the source corpus.

During the past decade, several areas of speech and language understanding have witnessed substantial breakthroughs from the use of data-driven models. In the area of dialogue systems, the trend is less obvious, and most practical systems are still built through significant engineering and expert knowledge. Nevertheless, several recent results suggest that data-driven approaches are feasible and quite promising. To facilitate research in this area, we have carried out a wide survey of publicly available datasets suitable for data-driven learning of dialogue systems. We discuss important characteristics of these datasets, how they can be used to learn diverse dialogue strategies, and their other potential uses. We also examine methods for transfer learning between datasets and the use of external knowledge. Finally, we discuss appropriate choice of evaluation metrics for the learning objective.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…Online Training Recent dialogue research has leveraged various data sources for training: corpora of humanhuman conversations in narrow domains (see Serban et al. (2018) and the list of currently available ParlAI tasks1 for a large set of available corpora), public conversations on internet…”
Towards Exploiting Background Knowledge for Building Conversation Systems (2018, background): “There has been an active interest in building datasets (Serban et al., 2015) for training dialog systems.”
A Knowledge-Grounded Neural Conversation Model (2017, background): “While these datasets (Serban et al. 2015) have grown dramatically in size thanks in particular to social media (Ritter, Cherry, and Dolan 2011), such datasets are still very far from containing discussions of every entry in Wikipedia, Foursquare, Goodreads, or IMDB.”
Learning Dense Representations for Entity Retrieval (2019, positive): “By contrast, since we are using a two-tower or dual encoder architecture (Gillick et al., 2018; Serban et al., 2018), our model cannot use any kind of attention over both mentions and entities at once, nor feature-wise comparisons as done by Francis-Landau et al.”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “mon approach is to train models to imitate humans in large corpora of crowdsourced or scraped conversations (Serban et al., 2015).”
Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “In modern research, the data used to train these models is often obtained from online chat platforms (Reddit, WeChat, Twitter, etc.) (Serban et al. 2015).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “A number of fine-tuning datasets are crowdsourced and publicly released for use by the research community (Serban et al., 2015; Huang et al., 2020), such as the ones we will use for training the BlenderBot 3 model in this work (see §3.”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, background): “Serban et al. (2015) list available corpora for training dialogue systems.”
Neural Approaches to Conversational AI (2018, positive): “(Serban et al., 2015) gave a good survey of public dialogue datasets that can used to develop dialogue agents.”
Learning End-to-End Goal-Oriented Dialog (2016, methodology): “Serban et al. (2015b) list all available corpora for training dialog systems, and unfortunately, no reliable resources exist to train and test end-to-end models in goal-oriented scenarios.”
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue (2016, background): “On the other hand, there has been a renewed interest in dialogue systems [29, 35], inspired by the success of datadriven approaches in other areas of natural language processing [10].”
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling (2018, background): “The first approach is to use a set of (often human generated) reference responses and compare a single generated response against them (Serban et al., 2015; Liu et al., 2016).”
On Evaluating and Comparing Conversational Agents (2018, background): “Serban et al. (2015) recently did a survey on available datasets for building and evaluating conversational dialogue systems, which illustrated another problem - there is a lack of high quality, open-ended, freely available conversational datasets.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “13Readers may refer to an old survey published in 2015, which covers datasets for both open-domain and task-oriented dialog models [118].”
Interpretation of Natural Language Rules in Conversational Machine Reading (2018, neutral): “Dialog The task we propose is, at its heart, about conducting a dialog (Weizenbaum, 1966; Serban et al., 2018; Bordes and Weston, 2016).”
A Dataset for Document Grounded Conversations (2018, background): “Serban et al. (2015) provide a comprehensive list of available datasets for building end-to-end conversational agents.”
Extending Neural Generative Conversational Model using External Knowledge Sources (2018, background): “Generative dialogue models (Sutskever et al., 2014; Serban et al., 2015) extends the language model learned by RNNs to generate natural language that are conditioned not only on the previous words generated in the response but also on a representation of the input context.”