The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus

By Aleksandra Piktus, Fabio Petroni, Yizhong Wang, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Ouguz, Edouard Grave and 2 more. 2021

79 citations recorded by the source corpus.

In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise. To this end, we propose a new setup for evaluating existing knowledge intensive tasks in which we generalize the background corpus to a universal web snapshot. We investigate a slate of NLP tasks which rely on knowledge - either factual or common sense, and ask systems to use a subset of CCNet - the Sphere corpus - as a knowledge source. In contrast to Wikipedia, otherwise a common background corpus in KI-NLP, Sphere is orders of magnitude larger and better reflects the full diversity of knowledge on the web. Despite potential gaps in coverage, challenges of scale, lack of structure and lower quality, we find that retrieval from Sphere enables a state of the art system to match and even outperform Wikipedia-based models on several tasks. We also observe that while a dense index can outperform a sparse BM25 baseline on Wikipedia, on Sphere this is not yet possible. To facilitate further research and minimise the community's reliance on proprietary, black-box search engines, we share our indices, evaluation metrics and infrastructure.

Subfields: Artificial Intelligence, Communication, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Improving Wikipedia verifiability with AI (2022, background): “To this aim, we rank the union of test citations in featured articles and test failed veriﬁcation citations.”
Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, methodology): “(2021); DPR + FiD (Piktus et al., 2021); KGI (Glass et al.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…the highest performance for question/answer pairs never seen during training ( i.e. , no overlap), suggesting a better ability to generalize to completely novel questions with novel answers (Piktus et al., 2021) 89.0 82.5 71.7 49.9 36.9 71.0 15.7 MT-DPR-WEB+FiD (Piktus et al., 2021) ( e.g. , 3 .”
PEER: A Collaborative Language Model (2022, methodology): “Otherwise, we augment these documents with ones from the Sphere corpus (Piktus et al., 2021), obtained using the pipeline described in (Petroni et al.”
Task-aware Retrieval with Instructions (2022, background): “Users’ intents can be diverse, requiring searching in an open-domain environment (Piktus et al., 2021), which is currently under-explored.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “Distributed-Faiss [269] provides the distributed solutions for ANNS, when the index is too large to fit into the memory of a single machine.”
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer (2022, background): “Recently, retrieval-augmented models are scaled up to very large corpora such as the web (Piktus et al., 2021; Borgeaud et al., 2021), making”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “On the other hand, Wikipedia and Wikidata are known to be great sources of general world knowledge and thus have been widely used by the community (Dinan et al., 2019; Thorne et al., 2018; Kwiatkowski et al., 2019; Piktus et al., 2021).”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “In many real-world situations, relevant evidence for a statement is not readily available, so some works have explored how to automatically retrieve evidence that may help support or refute a statement (Fan et al., 2020; Piktus et al., 2021), just as we do.”
Tool Learning with Foundation Models (2023, unknown): “Later works have gone beyond local repositories by leveraging the entire web as the knowledge source, which allows for improved temporal generalization and higher factual accuracy (Piktus et al., 2021; Lazaridou et al., 2022; Menick et al., 2022).”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “In many real-world situations, relevant evidence for a statement is not readily available, so some works have explored how to automatically retrieve evidence that may help support or refute a statement (Fan et al., 2020; Piktus et al., 2021), just as we do.”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, methodology): “The attribution corpora we consider are Sphere (Piktus et al., 2021) ( post_hoc_sphere_gpt4 ), which is a large static dump of CommonCrawl, and Google search results ( post_hoc_gs_gpt4 ).”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “Due to the diverse topics discussed in the questions, we use Sphere (Piktus et al., 2021)—a filtered version of Common Crawl 7 —as the corpus.”