Improving Passage Retrieval with Zero-Shot Question Generation

By Devendra Singh Sachan, Michael Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke S. Zettlemoyer. 2022

249 citations recorded by the source corpus.

We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage. This approach can be applied on top of any retrieval method (e.g. neural or keyword-based), does not require any domain- or task-specific training (and therefore is expected to generalize better to data distribution shifts), and provides rich cross-attention between query and passage (i.e. it must explain every token in the question). When evaluated on a number of open-domain retrieval datasets, our re-ranker improves strong unsupervised retrieval models by 6%-18% absolute and strong supervised models by up to 12% in terms of top-20 passage retrieval accuracy. We also obtain new state-of-the-art results on full open-domain question answering by simply adding the new re-ranker to existing models with no further changes.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “Query/Document Expansion A line of research which often involves autoregressive language models is that of document and query expansion.”
A Survey on Knowledge Distillation of Large Language Models (2024, unknown): “…benefit each component of the IR system, including the Query Rewriter (Srinivasan et al., 2022; Ma et al., 2023c), the Retriever (Dai et al., 2023b; Sachan et al., 2022, 2023; Schick and Sch¨utze, 2021; Meng et al., 2023; Peng et al., 2023b), and the Reranker (Bonifacio et al., 2022; Sun et al.,…”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “Strictly, as with Sachan et al. (2022), the only supervision resides in the LLM, in the processing of learning to follow instructions.”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, methodology): “, 2022) and UPR (Sachan et al., 2022), our approach employs task-specific few-shot adaption, while InPars and UPR’s prompts are task-independent and thus bear the same limitation of previous query generation approach (Ma et al.”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, methodology): “There are two popular methods, relevance generation (Liang et al., 2022) and query generation (Sachan et al., 2022; Drozdov et al., 2023).”
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “UPR (Sachan et al., 2022) and SGPT-CE (Muennighoff, 2022) introduce instructional query generation methods, while HELM (Liang et al., 2022) utilizes instructional relevance generation.”
Task-aware Retrieval with Instructions (2022, methodology): “…models to initialize our cross-encoder; we found that initializing TART-full with encoders of T5-based instruction-following pretrained models, namely T0-3B (Sanh et al., 2022) and FLANT5 (Chung et al., 2022), empirically leads to superior performance as found in prior work (Sachan et al., 2022).”
Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark (2024, methodology): “These retrieved passages are ranked by the LLM with two few-shot prompted LLM ranking functions: • Query Likelihood (QL) [108]: QL ( q, p ) = LLM ( q | p, P QL ) , where P QL is a prompt containing an instruction for judging query likelihood and several few-shot examples of relevant query and…”
LLM-confidence reranker: A training-free approach for enhancing retrieval-augmented generation systems (2026, unknown): “These rerankers are: • QLM (Sachan et al., 2022): A point-wise reranking approach that prompts pre-trained language models to generate a relevant query for each candidate document, then ranks based on the likelihood of the actual query, without fine-tuning.”