Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
By Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, S. Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui. 2024
165 citations recorded by the source corpus.
Large Language Models (LLMs) have demonstrated significant performance improvements across various cognitive tasks. An emerging application is using LLMs to enhance retrieval-augmented generation (RAG) capabilities. These systems require LLMs to understand user queries, retrieve relevant information, and synthesize coherent and accurate responses. Given the increasing real-world deployment of such systems, comprehensive evaluation becomes crucial. To this end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set), a high-quality evaluation dataset designed to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. While previous work has provided datasets and benchmarks to evaluate these abilities in isolation, FRAMES offers a unified framework that provides a clearer picture of LLM performance in end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions that require the integration of information from multiple sources. We present baseline results demonstrating that even state-of-the-art LLMs struggle with this task, achieving 0.40 accuracy with no retrieval. The accuracy is significantly improved with our proposed multi-step retrieval pipeline, achieving an accuracy of 0.66 (>50% improvement). We hope our work will help bridge evaluation gaps and assist in developing more robust and capable RAG systems.
Subfields: Artificial Intelligence, Information Systems, Retrieval-Augmented Generation, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- William Fedus
- Yi Tay
- Luke S. Zettlemoyer
- Yoshua Bengio
- Quoc V. Le
- Mostafa Dehghani
- Jianfeng Gao
- Percy Liang
In Evaluation & Benchmarks
Citations in the corpus
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025, unknown): “Additionally, DeepSeek-R1 excels on FRAMES, a long-context-dependent QA task, showcasing its strong document analysis capabilities.”
- Kimi K2: Open Agentic Intelligence (2025, unknown): “We benchmark the long-context capabilities on: MRCR 5 for long-context retrieval, and DROP [15], FRAMES [38] and LongBench v2 [2] for long-context reasoning.”
- Measuring short-form factuality in large language models (2024, background): “…benchmarks include LongFact (Wei et al., 2024), a benchmark of open-ended prompts, and FreshQA (Vu et al., 2023), a QA benchmark to evaluate performance on fast-changing knowledge, among other work (Lin et al., 2022a; Li et al., 2023; Cheng et al., 2023; Min et al., 2023; Zhao et al., 2024).”
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks (2024, unknown): “…retrieval and attributing (Kamradt, 2023; Kuratov et al., 2024; Song et al., 2025; Laban et al., 2024; Zhang et al., 2024b; Vodrahalli et al., 2024; Krishna et al., 2025), conversation (Bai et al., 2024a), coding (Liu et al., 2024a; Bogomolov et al., 2024), many-shot learning (Agarwal et al.,…”
- FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “Existing IR and RAG benchmarks are stale, evaluating on academic question answering datasets [20, 63, 61, 71, 86], or are not challenging, being constructed for RAG [87, 7, 40, 51, 71, 34].”