MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
By Yixuan Tang, Yi Yang. 2024
304 citations recorded by the source corpus.
Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. The MultiHop-RAG and implemented RAG system is publicly available at https://github.com/yixuantt/MultiHop-RAG/.
Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Information Retrieval & Search
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Retrieval & Search
Citations in the corpus
- Evaluation of Retrieval-Augmented Generation: A Survey (2024, methodology): “The searching utilizes these indexes to fetch relevant documents on the user’s query, often incorporating the optional rerankers [4,34,5,43] to refine the ranking of the retrieved documents.”
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, methodology): “Many benchmark datasets for open-domain question answering exist, including HotPotQA (Yang et al., 2018), MultiHop-RAG (Tang and Yang, 2024), and MT-Bench (Zheng et al., 2024).”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “RAG-for-math-QA [360] improves math question-answering by integrating a high-quality math text-book with retrieval-augmented generation, enhancing LLM-generated responses for middle-school algebra and geometry.”
- A Matryoshka Hierarchical RAG for Efficient Multi-Hop Question Answering (2026, unknown): “The application of RAGs to multi-hop QA [33] ranges from single-pass retrieval to iterative pipelines that interleave retrieval and reasoning.”
- ASTRA-QA: A Benchmark for Abstract Question Answering over Documents (2026, unknown): “Table 6 compares ASTRA-QA with representative RAG datasets [38, 39, 15, 40, 16] using three diversity metrics from prior text generation studies [41]: Q&A n-gram diversity (Q&A-NGD), answer cluster entropy (A-CE), and answer embedding homogeneity score (A-HS), together with average question and…”
- SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answering (2025, unknown): “The Q&A incorporate document metadata, representing complex, real-world scenarios often encountered in RAG applications 2 [24].”
- RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025, unknown): “To systematically assess the effectiveness of RAG and GraphRAG on general QA settings, we evaluate representative methods on widely used datasets and follow standard metrics used in prior work.”
- MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation (2026, unknown): “Research into MultiHop-RAG (Tang and Yang, 2024) and HotpotQA (Yang et al., 2018) has demonstrated that systems performing well on single-hop queries often fail significantly when required to connect multiple pieces of evidence.”