CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training

By Patrick Huber, Armen Aghajanyan, Barlas Oğuz, Dmytro Okhonko, Wen-tau Yih, Sonal Gupta, Xilun Chen. 2021

16 citations recorded by the source corpus.

With the rise of large-scale pre-trained language models, open-domain question-answering (ODQA) has become an important research topic in NLP. Based on the popular pre-training fine-tuning approach, we posit that an additional in-domain pre-training stage using a large-scale, natural, and diverse question-answering (QA) dataset can be beneficial for ODQA. Consequently, we propose a novel QA dataset based on the Common Crawl project in this paper. Using the readily available schema.org annotation, we extract around 130 million multilingual question-answer pairs, including about 60 million English data-points. With this previously unseen number of natural QA pairs, we pre-train popular language models to show the potential of large-scale in-domain pre-training for the task of question-answering. In our experiments, we find that pre-training question-answering models on our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “There are other large scale resources that could be considered to tackle KI tasks, such as large collections of question-answer pairs (Lewis et al., 2021a; Huber et al., 2021), structured knowledge sources (Berant et al.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “With the advent of web-scale QA datasets [68], QA-infused PLMs may provide signiicantly richer representation, enabling a wider range of applications.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, unknown): “CQADupStack [62] – – – 13,145 457,199 Quora Quora [63] – – 5,000 10,000 522,931 News ArchivalQA [64] 853,644 853,644 106,706 106,706 483,604 Web CCQA [65] 55M 130M – – –”