Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

By Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander H. Miller, Arthur Szlam, Jason Aaron Edward Weston. 2015

202 citations recorded by the source corpus.

A long-term goal of machine learning is to build intelligent conversational agents. One recent popular approach is to train end-to-end models on a large amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals & Le, 2015; Shang et al., 2015). However, this approach leaves many questions unanswered as an understanding of the precise successes and shortcomings of each model is hard to assess. A contrasting recent proposal are the bAbI tasks (Weston et al., 2015b) which are synthetic data that measure the ability of learning machines at various reasoning tasks over toy language. Unfortunately, those tests are very small and hence may encourage methods that do not scale. In this work, we propose a suite of new tasks of a much larger scale that attempt to bridge the gap between the two regimes. Choosing the domain of movies, we provide tasks that test the ability of models to answer factual questions (utilizing OMDB), provide personalization (utilizing MovieLens), carry short conversations about the two, and finally to perform on natural dialogs from Reddit. We provide a dataset covering 75k movie entities and with 3.5M training examples. We present results of various models on these tasks, and evaluate their performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…approaches to using long-term knowledge are either graph representations (Moon et al., 2019a) or unstructured text retrieval (Chen et al., 2017; Dodge et al., 2016; Dinan et al., 2019b), which is then prepended onto the dialogue history and attended over, an approach advocated by the memory…”
Wizard of Wikipedia: Knowledge-Powered Conversational agents (2018, methodology): “The work of Dodge et al. (2016) employed Memory Networks to perform dialogue discussing movies in terms of recommendation and open-ended discussion from Reddit, conditioning on a structured knowledge base.”
Improving Neural Language Models with a Continuous Cache (2016, background): “, 2010), they lack the capacity to adapt to their recent history, limiting their application to dynamic environments (Dodge et al., 2015).”
Learning Through Dialogue Interactions (2016, background): “Another line of research focuses on supervised learning for question answering from dialogues (Dodge et al., 2015; Weston, 2016), using either a given database of knowledge (Bordes et al.”
Dialog-based Language Learning (2016, methodology): “In machine learning, one line of research has focused on supervised learning from dialogs using neural models [22, 4].”
Visual Dialog (2016, background): “While some of the earliest developed chatbots were rule-based [56], end-to-end learning based approaches are now being actively explored [7,9,19,24,45,46,53].”
Personalizing Dialogue Agents: I have a dog, do you have pets too? (2018, methodology): “A promising direction, that is still in its infancy, to ﬁx this issue is to use a memory-augmented network instead (Sukhbaatar et al., 2015; Dodge et al., 2015) by providing or learning appropriate memories.”
Neural Approaches to Conversational AI (2018, neutral): “apter, their authors have created their own (subsets of) conversational data for training and testing, and then evaluated their systems against baselines and competing systems on these ﬁxed datasets. Dodge et al. (2016) used an existing dataset to deﬁne standard training and test sets, but it is rel”
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (2018, background): “otions to responses, end-to-end dialogue systems relying on neural networks and trained on conversation corpora (Shang et al., 2015; Vinyals & Le, 2015; Sordoni et al., 2015; Serban et al., 2015; Dodge et al., 2016; Mazare´ et al., 2018; Zhang et al., 2018) offer the promise of better generaliza”
Learning End-to-End Goal-Oriented Dialog (2016, methodology): “…contrast to other recent results on data-driven non -goal directed conversations, e.g. over dialogs on Twitter (Ritter et al. , 2011) or Reddit (Dodge et al. , 2016), where it was found that TF-IDF Match outperforms Nearest Neighbor, as general conversations on a given subject typically share…”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “…dialogue system architectures based on neural networks have shown promising results on several dialogue tasks (Ritter et al., 2011; Vinyals and Le, 2015; Lowe et al., 2015a; Sordoni et al., 2015b; Shang et al., 2015; Li et al., 2015; Serban et al., 2016; Serban et al., 2017b,a; Dodge et al., 2015).”
Training Millions of Personalized Dialogue Agents (2018, methodology): “As in (Dodge et al., 2015), we use a preexisting dump of REDDIT that consists of 1.”
Dialogue Learning With Human-In-The-Loop (2016, neutral): “Our work is related to the line of research that focuses on supervised learning for question answering (QA) from dialogues (Dodge et al., 2015; Weston, 2016), either given a database of knowledge (Bordes et al., 2015; Miller et al., 2016) or short texts (Weston et al., 2015; Hermann et al., 2015;…”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, methodology): “We compare IDS with several baselines: • IR: the basic tf-idf match model used in (Bor-des et al., 2016; Dodge et al., 2015).”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, background): “Our end-to-end approach would also be much more straightforward to integrate into a generalpurpose dialogue agent than one that relied on annotated dialogue states (Dodge et al., 2016).”
GuessWhat?! Visual Object Discovery through Multi-modal Dialogue (2016, methodology): “Many tasks have recently been introduced, including the bAbI tasks [42] for testing an agent’s ability to answer questions about a short story, the movie dialog dataset [11] to assess an agent’s capabilities regarding personal movie recommendation and a Wizardof-Oz framework [41] to evaluate an agen”