API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

By Minghao Li, Feifan Song, Yu Bowen, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li. Conference on Empirical Methods in Natural Language Processing, 2023

537 citations recorded by the source corpus.

Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/19c222d1f18317d58cc85491f37479bc0dc49f41

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Tool learning with large language models: a survey (2024, unknown): “In this survey, we conduct a systematic exploration of existing studies in two primary dimensions: (1) why tool learning is beneficial and (2) how tool learning is implemented.”
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (2024, background): “Additionally, API-centric datasets like API-Bank [26], ToolTalk [11], and ToolBench [60] assess agents’ capabilities to manipulate computer systems via APIs.”
ToolRL: Reward is All Tool Learning Needs (2025, unknown): “In addition, we present results on API-2 Bank (Li et al., 2023), a three-level evaluation framework comprising 73 diverse and complex API tools.”
TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, background): “…(Roy & Roth, 2015; Cobbe et al., 2021; Patel et al., 2021); tool-use evaluating agents’ proficiency in employing tools and reporting results (Li et al., 2023; Xu et al., 2023; Zhuang et al., 2023); and web navigation, testing agents’ ability to locate specific websites (Deng et al., 2023;…”
Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, methodology): “Hence, LLM-based agent has been studied and rapidly developed to understand and generate human-like instructions, facilitating sophisticated interactions and decision-making in a wide range of contexts [Yao et al. , 2023; Shinn et al. , 2023; Li et al. , 2023d].”