PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

By Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, Sebastian Riedel. 2021

275 citations recorded by the source corpus.

Abstract Open-domain Question Answering models that directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared with conventional models which retrieve and read from text corpora. QA-pair retrievers also offer interpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models fall short of the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text corpora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically generated QA-pairs. We introduce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outperform comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) while retaining high accuracy. Lastly, we demonstrate RePAQ’s strength at selective QA, abstaining from answering when it is likely to be incorrect. This enables RePAQ to “back-off” to a more expensive state-of-the-art model, leading to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling (2022, methodology): “Lewis et al. (2021) trained the RePAQ-retriever system on multiple tasks, but for their FiD defer mechanism used task-specific FiD checkpoints.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “We mark model that are also trained on additional synthetic data (Lewis et al., 2021c) with † .”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “We train DPR WEB by ﬁnetuning a PAQ-based (Lewis et al., 2021) bi-encoder check-point (O˘guz et al., 2021) for 40 epochs on 16 GPUs.”
You Only Need One Model for Open-domain Question Answering (2021, background): “UnitedQA (Cheng et al., 2021) and R2D2 (Fajcik et al., 2021) combine re-sults from an ensemble of extractive and generative readers, whereas PAQ (Lewis et al., 2021) directly retrieves answers with an FiD fallback.”
Learning to Retrieve Passages without Supervision (2021, methodology): “However, their approach utilizes PAQ (Lewis et al., 2021b), a dataset which was generated using models trained on NQ, and is therefore not unsupervised.”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “A long line of prior work has trained generative models for question answering (Du et al., 2017; Du and Cardie, 2018; Zhao et al., 2018; Lewis and Fan, 2019; Alberti et al., 2019; Puri et al., 2020; Yang et al., 2020; Bartolo et al., 2021; Lewis et al., 2021).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Thus, our dataset features a diverse collection of instructions (including human-written, template-based and LLM-generated), each paired with a human-written output.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “2M 0 21K Kinetics-700 [32] 0 542K 700 Moments in Time [57] 0 792K 339 Books&Wiki [93] 0 0 101M PAQ [41] 0 0 65M CC3M [68] 3.”
Towards Continual Knowledge Learning of Language Models (2021, methodology): “The process requires selecting answerable questions from a list of questions generated by the model introduced in Lewis et al. (2021) and converting them into cloze sentences.”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, methodology): “To this end, we propose using two corpora for retrieval pre-training: • A corpus of 65 million synthetically generated question-answer pairs from Wikipedia (PAQ, Lewis et al., 2021), which we target for open domain question answering and other passage retrieval tasks.”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, background): “Many approaches have been proposed to generate question-answer pairs given a passage (Du et al., 2017; Du and Cardie, 2018; Zhao et al., 2018; Lewis and Fan, 2019; Alberti et al., 2019; Puri et al., 2020; Lewis et al., 2021b).”
Boosting Search Engines with Interactive Agents (2021, neutral): “Another possible downside is the overlap between partitions, as pointed out in (Lewis et al., 2021a).”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “(2022) proposed a pipeline consisting of question generation (Ma et al., 2021; Lewis et al., 2021), negative mining and automatic labeling using large language models, and have shown it to be an effective alternative.”
Dialog Inpainting: Turning Documents into Dialogs (2022, methodology): “More recently, Ma et al. (2021) and Lewis et al. (2021) train special-purpose question generation models to synthetically generate millions of high-quality question answer pairs.”
Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering (2022, methodology): “For the answer generation component of QA systems, recent studies have used either extractive language models like BERT or generative language models like BART/GPT-2 (Min et al., 2021; Lewis et al., 2021).”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “• Dataset Augmentation Prior work on QA has performed data augmentation by either creating template-based or machine generated questions, e.g., for visual QA (Kaﬂe et al., 2017) and textual QA (Lewis et al., 2021 generally lack rich linguistic variations.”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, unknown): “…10,000 code_search (Husain et al., (He and McAuley, 2016) Asymmetric 100,000 S2ORC_title_abstract (Lo et al., 2020) Asymmetric 100,000 PAQ_pairs (Lewis et al., 2021) Asymmetric 25,000 wow (Dinan et al., 2019) Asymmetric 30,000 trex (Elsahar et al., 2018) Asymmetric 30,000 pubmed (Sen et al.,…”
GooAQ: Open Question Answering with Diverse Answer Types (2021, methodology): “Knowledge leakage across train and evaluation sets has been shown to significantly inflate performance numbers on recent open QA datasets (Lewis et al., 2021; Emami et al., 2020).”
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, unknown): “Our retrieval system is task agnostic, both in terms of domain and complexity, in contrast to previous approaches (Ein-Dor et al., 2020; Dai et al., 2022; Lewis et al., 2021).”