τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

By Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan. arXiv.org, 2024

820 citations recorded by the source corpus.

Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $\tau$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on<50% of the tasks, and are quite inconsistent (pass^8<25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.

Subfields: Artificial Intelligence, Management Information Systems

View Paper: https://www.semanticscholar.org/paper/70aa016c1f68fd5c0261f26ad20017b8307650af

Citations in the corpus

• Kimi K2: Open Agentic Intelligence (2025, neutral): “Recent benchmarks such as ACEBench [7] and τ -bench [86] have highlighted the importance of comprehensive tool-use evaluation, while frameworks like ToolLLM [59] and ACEBench [7] have demonstrated the potential of teaching models to use thousands of tools effectively.”
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, neutral): “5 in two aspects: TAU-bench [48] (including retail and airline domains) and Berkeley Function Call Leaderboard V3 (BFCL V3) [26], which measures the model’s ability to call user-defined functions to respond to users’ queries.”
• gpt-oss-120b&gpt-oss-20b Model Card (2025, neutral): “• Tool use : function calling ability with τ -Bench Retail [23], we provide the model with functions to call in the model’s developer message.”
• τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025, neutral): “…line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as customer…”
• A benchmark of expert-level academic questions to assess AI capabilities (2025, neutral): “The framing of the benchmark as "humanity's last exam" feels a bit strong to me.”
• Small Language Models are the Future of Agentic AI (2025, neutral): “…(an indicator of basic understanding), tool calling and code generation (both indicators of the ability to correctly communicate across the model → tool/code interface; see Figure 1; [74, 75]), and instruction following (ability to correctly respond back across the code ← model interface; [80]).”
• LLMs Corrupt Your Documents When You Delegate (2026, unknown): “…at work building benchmarks that better capture real-world work, building industry-specific benchmarks for customer service (Huang et al., 2024; Yao et al., 2024), enterprise knowledge work (Drouin et al., 2024; Xu et al., 2024), IT operations (Jha et al., 2025), or spanning multiple…”