ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

By Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian and 8 more. 2023

1,907 citations recorded by the source corpus.

Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.

Subfields: Artificial Intelligence, Large Language Models, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Maosong Sun
  3. Jianfeng Gao
  4. Percy Liang
  5. Sergey Levine
  6. Luke S. Zettlemoyer
  7. Zhiyuan Liu
  8. Quoc V. Le

In Large Language Models

  1. Maosong Sun
  2. Heng Ji
  3. Yankai Lin
  4. Zhilin Yang
  5. Zhi Jin
  6. Ji-rong Wen
  7. Wayne Xin Zhao
  8. Niklas Muennighoff

In LLM Agents & Tool Use

  1. Karthik Narasimhan
  2. Aleksandra Faust
  3. Maosong Sun
  4. Mustafa Safdari
  5. Izzeddin Gur
  6. Shunyu Yao

Citations in the corpus