Evaluation of Retrieval-Augmented Generation: A Survey
By Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, Zhaofeng Liu. 2024
292 citations recorded by the source corpus.
Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.
Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kalpesh Krishna
- Satyapriya Krishna
- Manaal Faruqui
- Anhad Mohananey
- Chris Dyer
- James R. Glass
- Yoshua Bengio
- Mohit Bansal
In Retrieval-Augmented Generation
In Evaluation & Benchmarks
Citations in the corpus
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, background): “However, despite this progress, the evaluation of RAG systems remains fragmented and insufficient, as existing benchmarks typically assess components like retrieval, factual correctness, and reasoning in isolation (Yu et al., 2024a).”
- Graph Retrieval-Augmented Generation: A Survey (2024, unknown): “How did the artistic movements of the 19th century impact the development of modern art in the 20th century?”
- Enhancing Domain-Specific Retrieval-Augmented Generation: Synthetic Data Generation and Evaluation using Reasoning Models (2025, unknown): “Traditional RAG evaluations focus primarily on document-level retrieval accuracy using metrics such as MRR@k or nDCG [2].”
- RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025, unknown): “…and hierarchical approaches use graph-inspired structures without fully specified entity–relation semantics: RAPTOR constructs hierarchical summary structures for multi-granular retrieval [32], while HippoRAG and its extensions build entity-linked graphs to guide chunk retrieval [10, 16].”
- ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generation (2025, unknown): “To enhance the trustworthiness and interpretability of LLMs, Retrieval-Augmented Generation (RAG) methods [15, 16, 24, 26, 71, 77, 80] have emerged as a core approach, which often retrieve relevant information from documents, relational data, and graph data to facilitate the QA tasks.”