RAGAs: Automated Evaluation of Retrieval Augmented Generation
By ES Shahul, Jithin James, Luis Espinosa Anke, Steven Schockaert. Conference of the European Chapter of the Association for Computational Linguistics, 2023
839 citations recorded by the source corpus.
We introduce RAGAs (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAGAs is available at [https://github.com/explodinggradients/ragas]. RAG systems are composed of a retrieval and an LLM based generation module. They provide LLMs with knowledge from a reference textual database, enabling them to act as a natural language layer between a user and textual databases, thus reducing the risk of hallucinations. Evaluating RAG architectures is challenging due to several dimensions to consider: the ability of the retrieval system to identify relevant and focused context passages, the ability of the LLM to exploit such passages faithfully, and the quality of the generation itself. With RAGAs, we introduce a suite of metrics that can evaluate these different dimensions without relying on ground truth human annotations. We posit that such a framework can contribute crucially to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/f5e9e5bbe22f0263be1f1ce88c66978a2b927772
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- LightRAG: Simple and Fast Retrieval-Augmented Generation (2024, positive): “Retrieval-Augmented Generation (RAG) systems have been developed to enhance large language models (LLMs) by integrating external knowledge sources (Sudhi et al., 2024; Es et al., 2024; Salemi et al., 2024).”
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, background): “LLMs have also shown promise at evaluating the performance of conventional RAG systems, automatically evaluating qualities like context relevance, faithfulness, and answer relevance (RAGAS, Es et al., 2023).”
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (2024, methodology): “In addition, recent advancements have been made in automated RAG evaluation tools, such as ARES (Saad-Falcon et al., 2023) and RAGAS (Es et al., 2023).”
- Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “Concurrently, state-of-the-art automated tools like RA-GAS [Es et al. , 2023], ARES [Saad-Falcon et al. , 2023], and TruLens 8 employ LLMs to adjudicate the quality scores.”
- Evaluation of Retrieval-Augmented Generation: A Survey (2024, methodology): “The interplay between the retrieval and generation components means that the entire system’s performance cannot be fully understood by evaluating each component in isolation [41,13].”
- SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answering (2025, unknown): “Retrieval-Augmented Generation Assessment (RAGAS) is used to compare baseline RAG to SemiRAG, it is a metric framework to evaluate RAG-based models, focusing on Answer Correctness, Similarity, and Relevance to ensure factual accuracy and contextual coherence [29].”
- RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation (2025, unknown): “However, these methods frequently face hallucination challenges (Maynez et al., 2020; Zhang et al., 2023b), where generated content deviates from retrieved information.”
- Chunking Is Not the Bottleneck: Why RAG Evaluation Must Look beyond Segmentation Strategy (2026, unknown): “We evaluate Hit@ { 1,3,5 } , MRR (Craswell, 2009), NDCG@5 (J¨arvelin and Kek¨al¨ainen, 2002), and Con-text Precision (Es et al., 2024), a rank-weighted precision over retrieved chunks that rewards configurations where relevant chunks appear earlier in the ranking.”