Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
By Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal. 2022
1,143 citations recorded by the source corpus.
Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Jianfeng Gao
- Jonathan Berant
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Mohit Bansal
- Graham Neubig
- Luc Van Gool
- Jonathan Berant
- Dacheng Tao
- Trevor Darrell
- Jianfeng Gao
- Xiaogang Wang
In Retrieval-Augmented Generation
- Ji-rong Wen
- Matei A. Zaharia
- Christopher Potts
- Omar Khattab
- Luyu Gao
- Jamie Callan
- Keshav Santhanam
- Ori Ram
In Prompting & Reasoning
- Graham Neubig
- Jonathan Berant
- Jonathan Herzig
- Zhengbao Jiang
- Ori Yoran
- Thomas Scialom
- Ramakanth Pasunuru
- Jane Dwivedi-Yu
Citations in the corpus
- Augmented Language Models: a Survey (2023, background): “Recent works (He et al., 2022; Trivedi et al., 2022) propose to combine a retriever with reasoning via chain-of-thoughts (CoT) prompting to augment a LM.”
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “, 2020) and has similarities to IRCoT (Trivedi et al., 2022).”
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, neutral): “Alternatively, LLMs can be prompted or trained to utilize tools, including search engines, as part of their reasoning process (Qu et al., 2025; Trivedi et al., 2022a).”
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, background): “Other systems have also combined these concepts for multi-document summarization (CAiRE-COVID, Su et al., 2020) and multi-hop question answering (ITRG, Feng et al., 2023; IR-CoT, Trivedi et al., 2022; DSP, Khattab et al., 2022).”
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “…the substantial advancements chain-of-thought prompting has brought to LLMs in multi-step reasoning Wei et al. (2022), numerous studies (Yao et al., 2022; Trivedi et al., 2023; He et al., 2023) Beyond multi-step reasoning tasks, Jiang et al. (2023) shift their emphasis to long-form generation.”
- Knowledge Graph Prompting for Multi-Document Question Answering (2023, methodology): “We use the default hyperparameters from their original GitHub repository to fine-tune these two LLMs. Prompting LLMs for MD-QA - Table 1 and 2 Following (Trivedi et al. 2022a), we randomly select questions from the development set for reporting the performance.”
- Active Retrieval Augmented Generation (2023, methodology): “…regarding when to retrieve is that LMs should retrieve information only when they lack the required knowledge to avoid unnecessary or inappropriate retrieval that occurs in passive retrieval augmented LMs (Khandelwal et al., 2020; Borgeaud et al., 2022; Ram et al., 2023; Trivedi et al., 2022).”
- A-MEM: Agentic Memory for LLM Agents (2025, neutral): “To enrich each memory note with meaningful context beyond its basic content and timestamp, we leverage an LLM to analyze the interaction and generate these semantic components.”
- A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, methodology): “RARR, SelfAsk and Interleaving are designed for retrieval settings only, while Inline and Check & Fix can be used in all settings.”
- Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, background): “2021; Trivedi et al. 2023; Li et al. 2023a), dialogue (Cai et al. 2019a,b; Peng et al. 2023), domain-specific question answering (Cui et al. 2023) and code generation (Zhou et al. 2023b).”
- Reasoning with Language Model Prompting: A Survey (2022, background): “, 2023b) Retrieval CoT GPT-3 175B few-shot prompt Explicit Knowledge IRCoT (Trivedi et al., 2022) Retrieval CoT with wiki Flan-T5/GPT-3 few-shot prompt”
- Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “In contrast, IRCOT [ Trivedi et al. , 2022 ] combines RAG with Chain of Thought (CoT) methodologies, alternating CoT-guided retrievals with retrieval-informed CoT processes, significantly boosting GPT-3’s performance across various question-answering tasks.”
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, methodology): “Following prior work (Khattab et al., 2022; Trivedi et al., 2023; Yoran et al., 2023), we evaluate on 500 random examples from the development set of each dataset.”
- Large Language Models for Information Retrieval: A Survey (2023, positive): “For example, when providing a passage about “Barack Obama”, language models may need additional knowledge about his university, which may not be included in the results of simply searching the initial query.”
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, methodology): “…capable of doing this based on the frequency of entities in queries (Mallen et al., 2023) or the generated outputs from models for multi-step QA (Trivedi et al., 2023), they are suboptimal: former methods are overly simplistic, failing to consider multi-hop queries; meanwhile, later ones are…”