Wizard of Wikipedia: Knowledge-Powered Conversational agents

By Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, Jason Aaron Edward Weston. 2018

1,031 citations recorded by the source corpus.

In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
Few-shot Learning with Retrieval Augmented Language Models (2022, background): “, 2018), dialogue (Dinan et al., 2019) or citation recommendation (Petroni et al.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2018b, 2019), error correction (Chollampatt and Ng, 2018), multilingual sentence embeddings (Artetxe and Schwenk, 2018), and dialogue (Miller et al., 2017; Dinan et al., 2019).”
LaMDA: Language Models for Dialog Applications (2022, neutral): “[96] that encompass a variety of topics.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “Wizard of Wikipedia (Dinan et al., 2019) 33.”
OPT: Open Pre-trained Transformer Language Models (2022, background): “On the Wizard-of-Internet dataset, which is fully unsupervised for all models, we see that OPT-175B obtains the lowest perplexity but still has lower UF1 than the models with Wizard-of-Wikipedia supervision.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, methodology): “…have already resulted in significant improvements in many NLP tasks: they have improved factual consistency in summarization models (Huang et al., 2020a), fidelity in translation (Bapna and Firat, 2019), and reliability in dialogue agents (Dinan et al., 2019; Fan et al., 2021; Zhang et al., 2020a).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…on several purpose-built dialogue datasets, including ones designed to teach consistent personas, knowledge, and empathy (Zhang et al., 2018; Dinan et al., 2018; Rashkin et al., 2019; Smith et al., 2020b; Roller et al., 2021), recall of past conversation details across multiple sessions…”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “This might be because the safety classiﬁer was trained to identify dialog utterances that are “ not OK to send in a friendly conversation with someone you just met online ”, which may encapsulate more than just toxic responses (Dinan et al., 2019b).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “, 2019) focuses on empathy, and Wizard of Wikipedia (Dinan et al., 2019c) focuses”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Furthermore, we have consistently found that domain-specific pretraining is important to high performance: that is, using dialoguelike pretraining significantly outperforms generic pretraining on resources like Wikipedia and BooksCorpus (Dinan et al., 2019b; Humeau et al., 2019; Dinan et al., 2019a,”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, methodology): “One speaker (termed the Wizard) is meant to be both engaging and knowledgeable on the topics, and has access to an information retrieval system over Wikipedia to supplement their own knowledge.”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “, 2018a,c], and Dialog [Dinan et al., 2019] (see Figure 1).”
Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, background): “…filling, and dialogue categories: HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), Natural Questions (NQ) (Kwiatkowski et al., 2019), T-REx (Elsahar et al., 2018), Zero Shot RE (zsRE) (Levy et al., 2017)), FEVER (Thorne et al., 2018), and Wizard of Wikipedia (WoW) (Dinan et al., 2018).”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “Surfacing knowledge from large corpora is a crucial step when dealing with knowledge intensive language tasks (Levy et al., 2017; Dinan et al., 2019; Elsahar et al., 2018; Petroni et al., 2021), such as open-domain question answering (Voorhees et al., 1999; Joshi et al., 2017; Yang et al., 2018;…”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “This was applied successfully to Wizard of Wikipedia (Dinan et al., 2019), QA (Lee et al., 2019a) and LIGHT tasks (Urbanek et al., 2019).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “Most existing research in KI-NLP uses Wikipedia as the source of knowledge (Kwiatkowski et al., 2019; Joshi et al., 2017; Thorne et al., 2018; Yang et al., 2018; Dinan et al., 2019; Petroni et al., 2021).”