BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
By Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeff Han, Isa Fulford, Hyung Won Chung, Alexandre Passos, William Fedus, Amelia Glaese. arXiv.org, 2025
491 citations recorded by the source corpus.
We present BrowseComp, a simple yet challenging benchmark for measuring the ability for agents to browse the web. BrowseComp comprises 1,266 questions that require persistently navigating the internet in search of hard-to-find, entangled information. Despite the difficulty of the questions, BrowseComp is simple and easy-to-use, as predicted answers are short and easily verifiable against reference answers. BrowseComp for browsing agents can be seen as analogous to how programming competitions are an incomplete but useful benchmark for coding agents. While BrowseComp sidesteps challenges of a true user query distribution, like generating long answers or resolving ambiguity, it measures the important core capability of exercising persistence and creativity in finding information. BrowseComp can be found at https://github.com/openai/simple-evals.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems
View Paper: https://www.semanticscholar.org/paper/41d1ea36a9af136efc42f3c85516d00cc1d13458
Citations in the corpus
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, unknown): “BrowseComp [45] measures the model’s ability as a web browsing agent to find correct answers for complicated questions.”
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (2025, unknown): “This example highlights that, while searching the large combinatorial space for a trip plan that satisfies all constraints is challenging, checking whether a given candidate solution satisfies these constraints is relatively straightforward. forces, Aider-Polyglot, AIME 2025, HMMT Feb 2025, HMMT Nov”
- Large Language Models for Information Retrieval: A Survey (2023, unknown): “It is challenging but necessary to consider how to build IR systems free from this category of bias.”
- Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems (2026, unknown): “BrowseComp (Wei et al., 2025) extends this design to 1,266 harder questions built by inverse authoring: annotators start from a known fact and add constraints until the answer cannot be found on the first page of five search-engine queries and cannot be solved by frontier models or by another…”