Generate rather than Retrieve: Large Language Models are Strong Context Generators

By W. Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, Meng Jiang. International Conference on Learning Representations, 2022

431 citations recorded by the source corpus.

Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, resulting in the generated documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b2542a738b75ee9b7ce1a13d8b78f9095d212412

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “These augmented LLMs can access real-time information aided by web search engines [34] and leverage domain-specific knowledge from external resources [54].”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, methodology): “Evaluation Criteria Following (Yu et al. 2022), we compute F1 and EM to compare the LLM’s answer and the ground-truth one.”
A Unified Encoder-Decoder Framework with Entity Memory (2022, methodology): “A popular approach to incorporate knowledge into the generation process is retrieving evidence documents from external sources (Lewis et al., 2020b; Izacard and 2021; Oguz et al., 2020; Yu et al., 2022c).”
Active Retrieval Augmented Generation (2023, methodology): “This shares a similar spirit with methods that use generated hypothetical titles or paragraphs from LMs as retrieval queries or evidences (Gao et al., 2022; Sun et al., 2022; Yu et al., 2022; Mao et al., 2021).”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, methodology): “Prompting has been applied to open-domain QA (Lazaridou et al., 2022; Sun et al., 2022; Yu et al., 2023) but its value in improving retrieval and QA for multi-step open-domain questions remains relatively underexplored.”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, methodology): “, 2022) further improve performance on diverse knowledge-intensive tasks, leveraging their strong parametric memories (Kandpal et al., 2022; Yu et al., 2022).”
REPLUG: Retrieval-Augmented Black-Box Language Models (2023, background): “While concurrent work (Mallen et al., 2022; Si et al., 2023; Yu et al., 2023; Khattab et al., 2022) has demonstrated that using a frozen retriever can improve GPT3 performance on open-domain question answering, we approach the problem in a more general setting, including language modeling and unders”
Knowledge Conflicts for LLMs: A Survey (2024, background): “However, intra-memory conflicts, where LLMs generate differing responses to similar inputs, undermine their reliability and utility by introducing undesirable uncertainty.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “It generates representations of both source code and retrieved contexts, then jointly processes the embeddings to complete the code.”
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2023, neutral): “More specifically, given a question q , we first use q verbatim to query a search engine, in our case G OOGLE S EARCH .”
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (2023, methodology): “However, most existing approaches primarily focus on exploiting LLMs for content generation (e.g., query or passage) rather than relevance ranking for groups of passages (Yu et al., 2023; Microsoft, 2023).”
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024, neutral): “Some recent work proposes using large language models as retrievers due to their ability to memorize extensive knowledge (Yu et al., 2022; Sun et al., 2022).”
Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System (2023, background): “By leveraging internal knowledge, LLMs can improve the performance of recommender systems without relying on external retrievers [23].”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, background): “Such parametric memory could help solve downstream tasks (Wang et al., 2021; Yu et al., 2023; Sun et al., 2023).”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, methodology): “For the 17% of questions where retrieval causes the LM to counterfactually answer correctly, re-call is especially high (0.88) compared to overall recall (0.42). terestingly, GenRead generally outperforms vanilla LMs despite relying on LMs’ parametric memory.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “While significant efforts have been undertaken in exploring their knowledge storage (Geva et al., 2021; Meng et al., 2022)and probing (Petroni et al., 2019; Zhong et al., 2021; Yu et al., 2023c), the exact mechanism by which LLMs capture the factual knowledge remains elusive.”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, neutral): “In this part, we will briefly review studies with these two types of generators: parameter-accessible (white-box) and parameter-inaccessible (black-box).”
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2022, background): “, 2022), sampling (Graves, 2012) or increased processing (Wang et al., 2022; Yu et al., 2022b).”
Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “Different from NLU methods, NLG methods are typically under the encoder-decoder generation framework, which poses unique challenges for leveraging knowledge into decoding the next tokens during generation [46, 48].”