Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

By Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal. 2022

1,143 citations recorded by the source corpus.

Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Graham Neubig
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Yejin Choi
  6. Jianfeng Gao
  7. Jonathan Berant
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Mohit Bansal
  2. Graham Neubig
  3. Luc Van Gool
  4. Jonathan Berant
  5. Dacheng Tao
  6. Trevor Darrell
  7. Jianfeng Gao
  8. Xiaogang Wang

In Retrieval-Augmented Generation

  1. Ji-rong Wen
  2. Matei A. Zaharia
  3. Christopher Potts
  4. Omar Khattab
  5. Luyu Gao
  6. Jamie Callan
  7. Keshav Santhanam
  8. Ori Ram

In Prompting & Reasoning

  1. Graham Neubig
  2. Jonathan Berant
  3. Jonathan Herzig
  4. Zhengbao Jiang
  5. Ori Yoran
  6. Thomas Scialom
  7. Ramakanth Pasunuru
  8. Jane Dwivedi-Yu

Citations in the corpus