Latent Retrieval for Weakly Supervised Open Domain Question Answering

By Kenton C. T. Lee, Ming-Wei Chang, Kristina Toutanova. 2019

1,173 citations recorded by the source corpus.

Recent work on open domain question answering (QA) assumes strong supervision of the supporting evidence and/or assumes a blackbox information retrieval (IR) system to retrieve evidence candidates. We argue that both are suboptimal, since gold evidence is not always available, and QA is fundamentally different from IR. We show for the first time that it is possible to jointly learn the retriever and reader from question-answer string pairs and without any IR system. In this setting, evidence retrieval from all of Wikipedia is treated as a latent variable. Since this is impractical to learn from scratch, we pre-train the retriever with an Inverse Cloze Task. We evaluate on open versions of five QA datasets. On datasets where the questioner already knows the answer, a traditional IR system such as BM25 is sufficient. On datasets where a user is genuinely seeking an answer, we show that learned retrieval is crucial, outperforming BM25 by up to 19 points in exact match.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Following Lee et al. [27] and Karpukhin et al. [22], we use the December 2018 dump.”
Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “Enabled by the modern neural network toolkits, researchers have started to explore an end-to-end training scheme that includes pre-training, retrieval and also answer extraction jointly (Lee et al., 2019; Guu et al., 2020).”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “We consider the following datasets, and use the same setting as Lee et al. (2019): • NaturalQuestions (Kwiatkowski et al., 2019) contains questions corresponding to Google search queries.”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “A deeper justification is given in Lee et al. (2019).”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “The inverse cloze task was proposed by Lee et al. (2019) to pre-train dense retrievers in an unsupervised way.”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “by (Lee et al., 2019; Min et al., 2019b,a; Asai et al., 2019) where the model is only required to produce a single normalized answer and 2) the standard multi-answer variant used with reading comprehension systems”
Improving language models by retrieving from trillions of tokens (2021, methodology): “Lee et al. (2019) use an inverse cloze task to ﬁnd semantic representations of passages for retrieval.”
Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “Lee et al. (2019) introduced the inverse cloze task for pre-training retrievers, which can then be ﬁne-tuned end-to-end on question-answering tasks.”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “For Natural Questions (NQ; Lee et al., 2019; Kwiatkowski et al., 2019) and TQA (Joshi et al.”
Big Bird: Transformers for Longer Sequences (2020, methodology): “Most prominently, SpanBERT [42], ORQA [54], REALM [34], RAG [57] have achieved strong performance for different tasks.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “) and CoQA (Reddy et al., 2019); datasets whose question writers did not see the answer text remain largely unsolved—for example, the Natural Questions (Kwiatkowski et al., 2019) and QuAC. Similarly, Lee et al. (2019) found that question answering datasets in which questions were written while annot”
Augmented Language Models: a Survey (2023, methodology): “…works augment LMs with a dense retriever by adding the retrieved documents to the current context (Chen et al., 2017; Clark and Gardner, 2017; Lee et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020; Zhong et al., 2022; Borgeaud et al., 2022;…”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “In contrast with ORQA, it can perform Graph-based, Entity-based and Text-match retrieval.”
How Context Affects Language Models' Factual Predictions (2020, methodology): “Finally, while we have chosen DrQA as our baseline to compare to recent work, there exist several more sophisticated supervised open-domain QA models that outperform it on a variety of open-domain QA tasks [Lee et al., 2019, Yang et al., 2019a, Guu et al., 2020].”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “Dense Retrieval (DR) using learned distributed representations is a promising direction to overcome this sparse retrieval bottleneck [9, 10, 11, 12, 13, 14, 15]: The representation space is fully learnable and can leverage the strength of pretraining, while the retrieval operation is sufﬁciently…”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “Comparing the no-clustering version of our method and ORQA, we see that using the generated data results in much better retrieval performance (more than 15% improvements on all metrics).”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “Recently, Lee et al. (2019) and Seo et al. (2019) introduce fully trainable models that retrieve a few candidates directly from large-scale Wikipedia collections.”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, methodology): “Other lines of research in open-domain QA include joint learning of retrieval and reader components (Lee et al., 2019) or direct phrase retrieval in a large collection of documents (Seo et al., 2019).”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, methodology): “Some models are trained with maximum marginal likelihood (MML) (Kadlec et al., 2016; Swayamdipta et al., 2018; Clark and Gardner, 2018; Lee et al., 2019), but it is unclear if it gives a meaningful improvement over the heuristics.”
AmbigQA: Answering Ambiguous Open-domain Questions (2020, methodology): “Following Lee et al. (2019), NQ-OPEN is constructed by ﬁltering N ATURAL Q UESTIONS to questions where at least one annotator provided a non-null short answer to the question.”