Promptagator: Few-shot Dense Retrieval From 8 Examples

By Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, Ming-Wei Chang. International Conference on Learning Representations, 2022

350 citations recorded by the source corpus.

Much recent research on information retrieval has focused on how to transfer from one task (typically with abundant supervised data) to various other tasks where supervision is limited, with the implicit assumption that it is possible to generalize from one task to all the rest. However, this overlooks the fact that there are many diverse and unique retrieval tasks, each targeting different search intents, queries, and search domains. In this paper, we suggest to work on Few-shot Dense Retrieval, a setting where each task comes with a short description and a few examples. To amplify the power of a few examples, we propose Prompt-base Query Generation for Retriever (Promptagator), which leverages large language models (LLM) as a few-shot query generator, and creates task-specific retrievers based on the generated data. Powered by LLM's generalization ability, Promptagator makes it possible to create task-specific end-to-end retrievers solely based on a few examples {without} using Natural Questions or MS MARCO to train %question generators or dual encoders. Surprisingly, LLM prompting with no more than 8 examples allows dual encoders to outperform heavily engineered models trained on MS MARCO like ColBERT v2 by more than 1.2 nDCG on average on 11 retrieval sets. Further training standard-size re-rankers using the same generated data yields another 5.0 point nDCG improvement. Our studies determine that query generation can be far more effective than previously observed, especially when a small amount of task-specific knowledge is given.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e86009d9f9b1cdf083a48d087552bc4153784451

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2023, methodology): “The filtering approach has been used in previous work to isolate high-quality synthetic queries (Dai et al., 2022; Saad-Falcon et al., 2023).”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “Repeating these EM-like steps until convergence can produce high-quality training sets [107].”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “Meanwhile, PROMPTAGA-TOR [Dai et al. , 2022] utilizes LLM as a few-shot query generator and, based on the generated data, creates task-specific retrievers.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Recently, Promptagator [153] leverages a large language model consisting of 137 billion parameters, i.”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, methodology): “There has been a strong recent interest in exploring information retrieval in general with LLMs based approaches, due to the importance of the applications and the power of LLMs to understand textual queries and documents (Dai et al., 2022; Tay et al., 2022b; Wang et al., 2023; Jagerman et al., 2023”
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “• Promptagator++ (Dai et al., 2023): A 110M cross-encoder re-ranker trained on pseudo queries generated by FALN 137B.”
Improving Text Embeddings with Large Language Models (2023, background): “For instance, Doc2query [32] and Promptagator [7] generate synthetic queries for unlabeled documents, which are then leveraged for document expansion or retriever training.”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “Similar techniques [42, 15, 23] have been widely used for removing dataset noises.”
A Survey of Large Language Models (2023, neutral): “like question answering [266], code synthesis [267], and information retrieval [268].”
ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings (2025, unknown): “In information retrieval specifically, methods like InPars [8, 7] and Promptagator [31] prompt LLMs with just a few example queries to generate large amounts of synthetic queries aligned with existing passages.”
Retrieval And Structuring Augmented Generation with Large Language Models (2025, unknown): “Embedding models have seen substantial improvements through specialized architectures [135, 185] and alignment fine-tuning [40, 291].”
Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation (2025, unknown): “…et al., 2022), LaPraDoR (Xu et al., 2022), SimLM (Wang et al., 2023)) fine-tune on the target documents themselves, while few-shot or parameter-efficient schemes (e.g., TSDAE (Wang et al., 2021a), adapters (Houlsby et al., 2019), prompt tuning (Dai et al., 2022)) require some labeled examples.”
Multilingual Information Retrieval with a Monolingual Knowledge Base (2025, unknown): “For positive pairs, existing work often uses a noisy data source to identify relevant queries, then applies some heuristic and consistency quality filters to improve data quality [15, 21, 22, 23].”
ReasonIR: Training Retrievers for Reasoning Tasks (2025, unknown): “In information retrieval, one family of approaches leverages LLMs to synthesize simple, relevant, and task-specific queries from documents based on few-shot exemplars (Bonifacio et al., 2022; Chaudhary et al., 2023; Dai et al., 2022).”
DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “Moreover, instruction-following LLMs have also shown better performance when directly prompted as rerankers (Ma et al., 2023; Sun et al., 2023).”