Benchmarking Large Language Models in Retrieval-Augmented Generation

By Jiawei Chen, Hongyu Lin, Xianpei Han, Le Sun. AAAI Conference on Artificial Intelligence, 2023

594 citations recorded by the source corpus.

Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/28e2ecb4183ebc0eec504b12dddc677f8aef8745

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2023, background): “We fine-tune our LLM judges to evaluate the RAG systems across three different capabilities, each of which are often analyzed by researchers and practitioners to gauge RAG system performance (Chen et al., 2023; James and Es, 2023): 1.”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “…2023a ] addresses this by training compressors at different granularities, while Long Context [ Xu et al. , 2023b ] and “Walking in the Memory Maze” [ Chen et al. , 2023a ] design summarization techniques to enhance LLM’s key information perception, particularly in dealing with extensive contexts.”
MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (2024, methodology): “However, existing RAG benchmarks, such as RGB (Chen et al., 2023) and RECALL (Liu et al., 2023), mainly evaluate a simple case where the answer of a query can be retrieved and solved using one single piece of evidence.”
Evaluation of Retrieval-Augmented Generation: A Survey (2024, methodology): “The searching utilizes these indexes to fetch relevant documents on the user’s query, often incorporating the optional rerankers [4,34,5,43] to refine the ranking of the retrieved documents.”
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, unknown): “But the retrieval of documents that only seem textually relevant—and are ultimately irrelevant, or “distracting”—negatively affects performance on general question-answering tasks (Cuconasu et al., 2024; Chen et al., 2024).”
RELATE: Relation Extraction in Biomedical Abstracts with LLMs and Ontology Constraints (2025, unknown): “While zero-shot LLMs (Li et al., 2025) can outperform fine-tuned models on smaller datasets, they struggle when confronted with complex, domain-specific tasks that demand specialized knowledge (Chen et al., 2024b; Ja-han et al., 2024).”
PyZoBot: A Platform for Conversational Information Extraction and Synthesis from Curated Zotero Reference Libraries through Advanced Retrieval-Augmented Generation (2024, unknown): “Following this, the system undertakes the task of methodically downloading each individual PDF document that is listed within the catalog.”
Chunking Is Not the Bottleneck: Why RAG Evaluation Must Look beyond Segmentation Strategy (2026, unknown): “Existing benchmarks—including BEIR (Thakur et al., 2021), RAGAS (Es et al., 2024), and recent RAG benchmarks (Chen et al., 2024)”