Measuring short-form factuality in large language models

By Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus. arXiv.org, 2024

332 citations recorded by the source corpus.

We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models"know what they know,"and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3f99d8e6dada94f5bcfc650be3aab7a24e39bab3

Citations in the corpus

• Large Language Models for Information Retrieval: A Survey (2023, unknown): “More severely, researchers [229, 230] show that some modules in IR systems such as retriever and reranker, especially those based on neural models, may prefer LLM-generated documents, since their topics are more consistent and the perplexity of them are lower compared with human-written documents.”
• Phi-4 Technical Report (2024, unknown): “We first report the values from OpenAI’s simple-evals benchmark, which is a framework (including prompts, temperature, and extraction) for evaluating MMLU [HBB + 20], GPQA diamond [RHS + 23], MATH [HBK + 21], HumanEval [CTJ + 21], MGSM [SSF + 22], and the SimpleQA [WKC + 24] F1-score.”
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, unknown): “To evaluate the model’s general abilities, we employed a set of widely-adopted open-source benchmark datasets, encompassing knowledge-intensive evaluations MMLU (EM) [14] and SimpleQA (Correct) [44], and instruction-following assessments IFEval (Prompt Strict) [52], SysBench (ISR) [29],”
• Kimi K2: Open Agentic Intelligence (2025, unknown): “Finally, general capabilities are assessed using MMLU [24], MMLU-Redux [18], MMLU-Pro [77], IFEval [91], Multi-Challenge [65], SimpleQA [79], and LiveBench [81] (as of 2024-11-25).”
• A benchmark of expert-level academic questions to assess AI capabilities (2025, unknown): “Therefore, we use calibration error specifically as a measure of a model's overconfidence, independently of its task performance, which can be interpreted as a form of hallucination.”
• Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “SimpleQA (Wei et al., 2024) contains 4,326 short-answer questions authored by human trainers, each verified by an independent second trainer and required to have a single, time-invariant answer.”