GAIA: a benchmark for General AI Assistants

By Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom. arXiv.org, 2023

1,001 citations recorded by the source corpus.

We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/ab8169d6e4dfabfe7c30ebec1bb871bf3e1551cd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Large Language Models for Information Retrieval: A Survey (2023, neutral): “We have organized existing approaches into distinct categories based on their functions: query rewriting, retrieval, reranking, and reader modules.”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “They cover mostly single-app tasks on a small set of domains (1–6 websites [64, 73, 24]) and evaluate agents on static test sets [59, 37], instead of across across varied conditions and inputs which are found in real-world scenarios.”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, background): “In line with the motivation of our study, concurrently, GAIA [49] introduces 466 questions that test fundamental abilities of models such as reasoning, multimodality handling, or tool use.”
Why Do Multi-Agent LLM Systems Fail? (2025, positive): “To further evaluate MAST ’s generalizability, we apply this pipeline to two additional MAS (Magentic-One (Fourney et al., 2024) and OpenManus (Liang et al., 2025)) and benchmarks (GAIA (Mialon et al., 2023) and MMLU (Hendrycks et al., 2020)) not used in the initial development of MAST .”
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (2024, background): “While our focus is on evaluating LLM capabilities, other evaluation dimensions, such as safety [19], fairness [8], agentic planning [18, 23, 16], and hallucination detection [24, 25, 11], are equally important.”
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025, neutral): “…of searching (Joshi et al., 2017; Yang et al., 2018; Thorne et al., 2018; Dinan et al., 2019; Lee et al., 2019; Fan et al., 2019; Petroni et al., 2021; Yao et al., 2022; Mialon et al., 2023), BrowseComp focuses on the ability to retrieve hard-to-find, deeply entangled information on the web.”
WebThinker: Empowering Large Reasoning Models with Deep Research Capability (2025, positive): “• GAIA [33]: A benchmark dataset designed to evaluate the capabilities of general artificial intelligence (AGI).”
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, positive): “Mialon et al. (2023) proposed GAIA, a benchmark for general AI assistants that incorporates multi-step interactions and uses simplified answer formats.”
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (2024, neutral): “GAIA (Mialon et al., 2023) focuses on agent interactions with the real world, providing 466 questions that are conceptually simple for humans but challenging for current AI systems.”
Competition-Level Problems are Effective LLM Evaluators (2023, neutral): “1 https://twitter.com/keirp1/status/ 1724518513874739618 2 0 1 0 2 0 1 2 2 0 1 4 2 0 1 6 2 0 1 8 2 0 2 0 2 0 2 2 2 0 2 4 Problem Release Date how to accurately evaluate the reasoning abilities of LLMs and the extent of data contamination issues (Mialon et al., 2023; Zhou et al., 2023).”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “Existing evaluation frameworks consider either holistic evaluations [185, 186] or benchmark the models for weak spots such as task framing [187, 188], long temporal dependencies, invalid formatting or refusal to follow instructions [189].”
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, methodology): “The tasks in GAIA are not annotated with specific websites, we instruct the agent to search for answers starting from Google Search.”
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024, background): “…in vision-language models (VLMs) which can process image-text inputs to generate text, used for many multimodal tasks such as image captioning (Vinyals et al., 2015), visual question answering (Antol et al., 2015), and other benchmarks (Mialon et al., 2023; Yue et al., 2023; Tong et al., 2024).”
OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024, positive): “GAIA consists of 466 curated tasks across three levels.”
AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction (2026, unknown): “Related efforts such as GAIA (Mialon et al., 2023) and BFCL (Patil et al., 2025) evaluate the ability of agents to plan multi-step actions and invoke external tools or function calls.”
Multi-View Encoders for Performance Prediction in LLM-Based Agentic Workflows (2025, unknown): “Large language models (LLMs) have catalyzed the development of agentic systems capable of executing complex, multi-step tasks autonomously (Hong et al., 2024; Wu et al., 2024; Xi et al., 2024; Mialon et al., 2024).”