Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval

By Omar Khattab, Christopher Potts, Matei A. Zaharia. 2021

82 citations recorded by the source corpus.

Multi-hop reasoning (i.e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge. To retrieve evidence passages, multi-hop models must contend with a fast-growing search space across the hops, represent complex queries that combine multiple information needs, and resolve ambiguity about the best order in which to hop between training passages. We tackle these problems via Baleen, a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals in the many-hop setting. To tame the search space, we propose condensed retrieval, a pipeline that summarizes the retrieved passages after each hop into a single compact context. To model complex queries, we introduce a focused late interaction retriever that allows different parts of the same query representation to match disparate relevant passages. Lastly, to infer the hopping dependencies among unordered training passages, we devise latent hop ordering, a weak-supervision strategy in which the trained retriever itself selects the sequence of hops. We evaluate Baleen on retrieval for two-hop question answering and many-hop claim verification, establishing state-of-the-art performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Information Retrieval & Search, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Retrieval-Augmented Generation

In Information Retrieval & Search

In Question Answering & Reading Comprehension

Citations in the corpus

Cut the CARP: Fishing for zero-shot story evaluation (2021, methodology): “Rather than being solely local critiques of stories, perhaps there is a method similar to Khattab et al. (2021) where one can perform critique as a multi-hop inference method, and accumulate critiques between hops.”
Retrieval-Enhanced Machine Learning (2022, methodology): “[33] used a retrieval component for multi-hop reasoning, where the retrieved facts from each hop are summarized into a short context and becomes a part of the query for the subsequent hops.”
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (2022, methodology): “In the retrieval-augmented NLP literature, multi-hop search (Xiong et al., 2020; Khattab et al., 2021a) and con-D versational search (Del Tredici et al., 2021; Raposo et al., 2022) pipelines have received much attention.”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “…but interest has been growing in building multi-stage pipelines and agents that decompose complex tasks into more manageable calls to LMs in an effort to improve performance (Qi et al., 2019; Khattab et al., 2021a; Karpas et al., 2022; Dohan et al., 2022; Pourreza & Rafiei, 2023; Yao et al., 2023).”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, methodology): “Xiong et al. (2021) extends the widely-used Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) to multihop setting, which has since been improved by Khattab et al. (2021).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “More recently, without using hyperlinks, multi-hop dense retrieval [331], [332] has been proposed to enhance the performance of complex QA tasks [57].”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, methodology): “We are also in-601 spired by methods that iteratively retrieve knowl-602 edge from explicit knowledge sources for multi-603 step reasoning, such as (Xiong et al., 2021; Qi et al., 2019; Khattab et al., 2021).”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, methodology): “We are also inspired by methods that iteratively retrieve knowledge from explicit knowledge sources for multi-step reasoning, such as (Xiong et al., 2021; Qi et al., 2019; Khattab et al., 2021; Mo et al., 2022b).”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, methodology): “Next, we evaluate Maize outside the training domain using BEIR (Thakur et al., 2021), Wikipedia Open QA as in Khattab et al. (2021b), and LoTTE.”