ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
By Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian and 8 more. 2023
1,907 citations recorded by the source corpus.
Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.
Subfields: Artificial Intelligence, Large Language Models, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Maosong Sun
- Jianfeng Gao
- Percy Liang
- Sergey Levine
- Luke S. Zettlemoyer
- Zhiyuan Liu
- Quoc V. Le
In Large Language Models
In LLM Agents & Tool Use
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, background): “…domains and tasks such as program repair (Zheng et al., 2024; Olausson et al., 2023), optimization (Madaan et al., 2023a), test generation (Steenhoek et al., 2023), documentation generation (Luo et al., 2024), tool usage (Patil et al., 2023; Qin et al., 2024), SQL (Sun et al., 2023), and more.”
- CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, background): “Examples such as ToolFormer (Schick et al., 2023), HuggingGPT (Shen et al., 2023), Gorilla (Patil et al., 2023), ToolLlama (Qin et al., 2023) and ToolCoder (Zhang et al., 2023a) demonstrate LLMs’ proficiency in tool utilization.”
- Cognitive Architectures for Language Agents (2023, background): “It would be powerful to extend such schemes to more complicated tasks with grounding (Qin et al., 2023) and long-term memory; enable learning of decision making procedures; and reduce the cost of extensive reasoning to facilitate more deliberate planning (potentially via smaller task-specific models”
- TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, background): “These agents have shown interesting capabilities, such as tool use (Schick et al., 2023; Patil et al., 2023; Qin et al., 2024) and various forms of reasoning (Wei et al., 2022; Yao et al., 2022; Lewkowycz et al., 2022), potentially fulfilling the role of some of the cognitive substrates that were…”
- Understanding HTML with Large Language Models (2022, methodology): “ToolAlpaca (Tang et al., 2023) and TOOLLLM (Qin et al., 2023) are two examples that study multi-step tool-use with iterated API calls.”
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, positive): “The recent advancement of large language models (LLMs), such as ChatGPT and GPT-4 (OpenAI, 2023), have sparked significant interest in developing LLM-based autonomous agents (AutoGPT, 2022) for complex task execution (Qin et al., 2023; Schick et al., 2023).”
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024, background): “A number of related works have proposed augmenting language models with tools at test-time, which can greatly improve their performance on downstream tasks [11, 26, 27].”
- ChatDev: Communicative Agents for Software Development (2023, background): “…(Park et al., 2023; Sumers et al., 2023), planning (Chen et al., 2023b; Liu et al., 2023) and tool use (Schick et al., 2023; Cai et al., 2023; Qin et al., 2023a; Ruan et al., 2023; Yang et al., 2023b), enabling them to reason in complex scenarios (Wei et al., 2022b; Zhao et al., 2023; Zhou…”
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, neutral): “Kinniment et al. (2023) introduced a small but challenging agent benchmark, testing open-ended capabilities that push the limits of current frontier LLMs. Qin et al. (2024) explored multi-step function calling conversations with LLM-based grading.”
- τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025, neutral): “Following a long line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as…”
- Executable Code Actions Elicit Better LLM Agents (2024, methodology): “1, an LLM using with CodeAct (top right) can apply the same sequence of tools (e.g., passing one tool’s output as input to another tool using the data flow feature) to all inputs through for-loops (i.e., control flow feature) with one action; while text or JSON have to take action for every input (t”
- AgentTuning: Enabling Generalized Agent Abilities for LLMs (2023, neutral): “In addition, many efforts are dedicated to improving LLMs in specific aspects, involving fine-tuning the LLMs using datasets tailored to specific tasks (Deng et al., 2023; Qin et al., 2023; Chen et al., 2023).”
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “Ideally, automatic evaluation of the LLMs’ generation results can be easily realized with ground truths from existing datasets, if available (Zhu et al., 2023).”
- Kimi K2: Open Agentic Intelligence (2025, positive): “Recent benchmarks such as ACEBench [7] and τ -bench [86] have highlighted the importance of comprehensive tool-use evaluation, while frameworks like ToolLLM [59] and ACEBench [7] have demonstrated the potential of teaching models to use thousands of tools effectively.”
- Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, positive): “…(despite being not universally applicable in all possible environments), including programming (HumanEval (Chen et al., 2021)), web search (WebShop (Yao et al., 2022)), text-based manipulation tasks (Alfworld (Shridhar et al., 2020)), and LMs with tool use (ToolBench (Qin et al., 2024)).”
- ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2023, background): “Tool Use The utilization of external tools in LLMs has emerged as a rapidly growing research area (Mialon et al. 2023; Qin et al. 2023a).”