Generation-Augmented Retrieval for Open-Domain Question Answering
By Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Weizhu Chen. 2020
323 citations recorded by the source corpus.
We propose Generation-Augmented Retrieval (GAR) for answering open-domain questions, which augments a query through text generation of heuristically discovered relevant contexts without external resources as supervision. We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. Moreover, as sparse and dense representations are often complementary, GAR can be easily combined with DPR to achieve even better performance. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader, and consistently outperforms other retrieval methods when the same generative reader is used.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Luke S. Zettlemoyer
- Graham Neubig
- Percy Liang
- Hannaneh Hajishirzi
- Yejin Choi
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Jianfeng Gao
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Xiaogang Wang
- Andrew Zisserman
- Ming-Hsuan Yang
In Retrieval-Augmented Generation
In Natural Language Processing
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Mohit Bansal
- Sebastian Riedel
- Wen-tau Yih
- Christopher D. Manning
- Jason Aaron Edward Weston
Citations in the corpus
- Improving Wikipedia verifiability with AI (2022, methodology): “The sparse retrieval sub-system uses a seq2seq model (Lewis et al., 2019; Mao et al., 2020) to translate the citation context into query text , and then matches the resulting query—a sparse bag-of-words vector—on a BM25 index (Robertson et al., 1995; Baeza-Yates et al., 1999; Manning et al., 2008;…”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, methodology): “5 points better than GAR on average).”
- You Only Need One Model for Open-domain Question Answering (2021, methodology): “We also compare our pre-training to that of the additional data augmentation (Mao et al., 2021a; Ren et al., 2021; Oguz et al., 2021).”
- Memory-assisted prompt editing to improve GPT-3 after deployment (2022, methodology): “Our approach is inspired and supported by the recent success of generate and retrieve (Mao et al., 2021) methods.”
- What Makes Good In-Context Examples for GPT-3? (2021, background): “, 2018), QA (Karpukhin et al., 2020; Mao et al., 2020), dialogue generation (Yan et al.”
- Transformer Memory as a Differentiable Search Index (2022, background): “Alternatively, generation could be used to augment a query (Mao et al., 2020).”
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “Recently, researchers have utilized deep learning to improve traditional passage retrievers, including document expansions (Nogueira et al., 2019c), question expansions (Mao et al., 2020) and term weight estimation (Dai and Callan, 2019).”
- Condenser: a Pre-training Architecture for Dense Retrieval (2021, methodology): “GAR uses a learned deep LM BART (Lewis et al., 2020) to expand queries.”
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023, background): “More-over, some studies have investigated utilizing the model’s predictions as supplementary context to inform the retrieval process (Mao et al., 2020; Li et al., 2022; Wang et al., 2023; Zemlyanskiy et al., 2022).”
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023, background): “A closely related work called G AR (Mao et al., 2021) augments queries with generated background information.”
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, background): “For example, our community summaries are a kind of self-memory (Selfmem, Cheng et al., 2024) for generation-augmented retrieval (GAR, Mao et al., 2020) that facilitates future generation cycles, while our parallel generation of community answers from these summaries is a kind of iterative…”
- LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “2 In this work, inspired by dense retrieval in text retrieval domain (Guu et al., 2020; Karpukhin et al., 2020; Xiong et al., 2020; Mao et al., 2020; Lewis et al., 2020), we propose a more efficient attention-less framework.”
- Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “However, the previous work only used generation for query expansion (Mao et al., 2021), which did not exploit the potential of directly generating contextual documents for open-domain questions.”
- Reader-Guided Passage Reranking for Open-Domain Question Answering (2021, methodology): “We take the top retrieved passages of GAR (Mao et al., 2020) on Trivia and its combination with DPR (Karpukhin et al., 2020) out reranking and used for final passage reading in R3, which represents an apple-to-apple comparison to the R2 architecture without any additional information but higher-quali”
- Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, methodology): “DDRQA [84] Pipeline Iterative Extractive GAR [85] Pipeline Iterative Extractive Rule-based”
- Understanding the planning of LLM agents: A survey (2024, background): “Retrieval Augmented Generation (RAG) [ Lewis et al. , 2020; Mao et al. , 2020; Cai et al. , 2022] techniques are proposed to aid text generation with retrieved information.”