FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

By Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc V. Le and 1 more. 2023

394 citations recorded by the source corpus.

Most large language models (LLMs) are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality of LLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Through human evaluations involving more than 50K judgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search engine into the prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that both the number of retrieved evidences and their order play a key role in influencing the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we release FreshQA at github.com/freshllms/freshqa and commit to updating it at regular intervals.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Large Language Models

Citations in the corpus

Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, background): “This approach mitigates issues with frequent manual updates required for maintaining previous datasets (Vu et al., 2023; Kasai et al., 2024).”
ToolRL: Reward is All Tool Learning Needs (2025, positive): “…to enhance the capabilities of LLMs. Early studies introduced the concept of equipping LLMs with external tools to overcome their inherent limitations (Schick et al., 2023; Qin et al., 2024b; Yao et al., 2023), such as program executors (Chen et al., 2022) and search engines (Vu et al., 2023).”
RAFT: Adapting Language Model to Domain Specific RAG (2024, background): “Indeed, adapting LLMs to the specialized domains (e.g., recent news, enterprise private documents, or program resources constructed after the training cutoff) is essential to many emerging applications (Vu et al., 2023; Lazaridou et al., 2022) and is the focus of this work.”
Knowledge Conflicts for LLMs: A Survey (2024, positive): “Retrieval-augmented generation (RAG) fetches relevant documents from external sources to supplement the model’s knowledge without altering its parameters (Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020; Lazaridou et al., 2022; Vu et al., 2023).”
A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models (2024, background): “It helps the model follow human commands for specific tasks (Wang et al., 2023; Chung et al., 2022; Iyer et al., 2023; Sun et al., 2023b) and eventually increases the faithfulness of the model’s outputs.”
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, methodology): “To reduce the difficulty of annotation, we selected only those questions related to daily life, and preserved only three retrieved passages for each question.”
Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024, background): “For example, factual-centred metrics [19, 20, 40, 57] and benchmarks [34, 35, 65] have been proposed to measure and reduce hallucination on specific datasets.”
A Survey on Evaluation of Large Language Models (2023, neutral): “To assess the ability of LLMs in dynamic QA about current world knowledge, Vu et al. [186] introduced FRESHQA.”
Tool learning with large language models: a survey (2024, positive): “With the continuous enhancement of LLMs capabilities, it is expected that LLMs will become proficient in using tools to solve complex problems as human [16], a concept known as tool learning with LLMs. Tool learning emerges as a promising solution to mitigate these limitations of LLMs by enabling…”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “A limited number of datasets refresh over time to avoid LLM decontamination [32, 80, 67, 84], however, these contain easy and unrealistic questions.”