Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

By Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Leo Wright Hall, Percy Liang, Christopher Potts, Matei A. Zaharia. arXiv.org, 2022

389 citations recorded by the source corpus.

Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple"retrieve-then-read"pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37-120%, 8-39%, and 80-290% relative gains against the vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively. We release DSP at https://github.com/stanfordnlp/dsp

View Paper: https://www.semanticscholar.org/paper/03532123ccffae8d411264320e8a5ae2b6eddea0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Percy Liang
2. Mohit Bansal
3. Jianfeng Gao
4. Yi Tay
5. Sewon Min
6. Luke S. Zettlemoyer
7. Zhe Gan
8. Jonathan Berant

Citations in the corpus

• Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “As for ”how to use” the retrieved data, integration techniques have been developed across various levels of the model architecture, including the input [Khattab et al. , 2022], intermediate [Borgeaud et al. , 2022], and output layers [Liang et al. , 2023].”
• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2023, unknown): “10 We retrieve all of the search results, including the answer box , organic results , and other useful information, such as the knowledge graph , questions and answers from crowdsourced QA platforms, and related questions that search users also ask (see Figure 6 in Appendix F).”
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “In these sequential training methods, the process typically begins with the independent pretraining of either the retriever or the generator, after which the pretrained module is fixed while the other module undergoes training.”
• Large Language Models as Analogical Reasoners (2023, background): “…with more structured algorithms and search methods (Khot et al., 2022; Drozdov et al., 2022; Zelikman et al., 2022; Yao et al., 2023; Press et al., 2022; Khattab et al., 2022; Jung et al., 2022), as well as longer-horizon action and planning (Yao et al., 2022; Hao et al., 2023; Park et al., 2023).”
• REPLUG: Retrieval-Augmented Black-Box Language Models (2023, background): “While concurrent work (Mallen et al., 2022; Si et al., 2023; Yu et al., 2023; Khattab et al., 2022) has demonstrated that using a frozen retriever can improve GPT3 performance on open-domain question answering, we approach the problem in a more general setting, including language modeling and unders”
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, background): “Specifically, Khattab et al. (2022), Press et al. (2023), Pereira et al. (2023) and Khot et al. (2023) proposed to first decompose the multi-hop queries into simpler single-hop queries, repeatedly access the LLMs and retriever to solve these sub-queries, and merge their solutions to formulate a…”
• RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, methodology): “…2020; Borgeaud et al., 2022), recent work has suggested that simply adding retrieved documents to the input of a standard LM can also work well (Khattab et al., 2022; Ram et al., 2023; Shi et al., 2023), thus making it possible to use retrieval-augmented strategies in combination with LLMs…”
• Cognitive Architectures for Language Agents (2023, background): “• Mixing language-based reasoning and code-based planning may offer the best of both worlds.”
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “…Qiao et al., 2024), or to use specific tools or memory (Yao et al., 2022; Lanchantin et al., 2024; Schick et al., 2024), while others investigate how one may distill the reasoning from a large language model into a smaller language model Ho et al. (2022); Li et al. (2022); Hsieh et al. (2023).”
• DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “…lightweight finetuning techniques—for adapting LMs to new tasks (Kojima et al., 2022), eliciting systematic reasoning from them (Wei et al., 2022; Wang et al., 2022b), and augmenting them with retrieved sources (Guu et al., 2020; Lazaridou et al., 2022) or with tools (Yao et al., 2022).”
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, methodology): “Following prior work (Khattab et al., 2022; Trivedi et al., 2023; Yoran et al., 2023), we evaluate on 500 random examples from the development set of each dataset.”
• Generative Agents: Interactive Simulacra of Human Behavior (2023, background): “Recent studies have attempted to go beyond frst-order prompting by augmenting language models with a static knowledge base and an information retrieval scheme [53] or with a simple summarization scheme [104].”
• A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, unknown): “A related strategy is Demonstrate-Search-Predict (Khattab et al., 2023).”
• Active Retrieval Augmented Generation (2023, background): “Some works in multihop QA decompose the full question into sub-questions, each of which is used to retrieve extra information (Press et al., 2022; Yao et al., 2022; Khot et al., 2022; Khattab et al., 2022).”
• Large Language Models for Information Retrieval: A Survey (2023, unknown): “To allow LLMs to actively use search engines, Self-Ask [188] and DSP [189] try to employ few-shot prompts for LLMs, triggering them to search queries when they believe it is required.”
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, background): “Other systems have also combined these concepts for multi-document summarization (CAiRE-COVID, Su et al., 2020) and multi-hop question answering (ITRG, Feng et al., 2023; IR-CoT, Trivedi et al., 2022; DSP, Khattab et al., 2022).”
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023, background): “Khattab et al. (2022) proposed a DSP programming framework that supports various retrieval-augmented methods.”
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, methodology): “Lastly, it will be useful to assess whether DSP, which is hardcoded for 2-hop questions like that of HotpotQA, will work well for a dataset with a varied number of hops like that of MuSiQue.”