GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

By Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, Ying Shan. Neural Information Processing Systems, 2023

329 citations recorded by the source corpus.

This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at https://github.com/StevenGrove/GPT4Tools.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b458fc5261595f44b36325e5eaea1f874d65138f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, background): “The recent advancements in artificial intelligence have focused on conversational assistants [42, 31, 30, 13, 17] that possess a strong ability to understand user intentions [35] and then execute actions [5, 51].”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “There are also re-searches [57, 60] that utilize the reasoning ability of LLMs and invoke pre-defined tools for visual applications.”
ChatDev: Communicative Agents for Software Development (2023, background): “…2023), planning (Chen et al., 2023b; Liu et al., 2023) and tool use (Schick et al., 2023; Cai et al., 2023; Qin et al., 2023a; Ruan et al., 2023; Yang et al., 2023b), enabling them to reason in complex scenarios (Wei et al., 2022b; Zhao et al., 2023; Zhou et al., 2023a; Ma et al., 2023; Zhang…”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “By training ImageBind-LLM on the GPT4Tools dataset using our proposed training paradigm, we observe its impressive proficiency in calling different APIs, enabling it to accomplish a wide range of tasks, even when encountering previously unseen tools.”
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2023, methodology): “Concurrent with our work, GPT4Tools (Yang et al. 2023) fine-tuning compact models to incorporate multi-modal tools, which concentrates on a”
LISA: Reasoning Segmentation via Large Language Model (2023, background): “Moreover, numerous works [32, 44, 49, 52, 54] utilize prompt engineering, connecting independent modules via API calls, but without the benefits of end-to-end training.”
Agentic AI: Autonomous Intelligence for Complex Goals—A Comprehensive Survey (2025, unknown): “For example, classical AI was never focused on image recognition or language translations [6], [7] but instead aimed at achieving very narrow goals.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “…LLMs with various functioning agents where language acts as general interface (Wu et al., 2023; Gupta & Kembhavi, 2023; Yang et al., 2023b; Liang et al., 2023; Shen et al., 2023; Yang et al., 2023a; Surís et al., 2023), and remarkable success has been demonstrated in such plugin-style frameworks.”
A Comprehensive Overview of Large Language Models (2023, unknown): “Rather than storing information locally, various methods retrieve query-related context through a web search and forward it to LLMs [258], [259], [147].”
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (2024, methodology): “LLM-based agents Li et al. [2023], Liu et al. [2023a,b,c], Shen et al. [2023], Wu et al. [2023], Yang et al. [2023a], Shen et al. [2024], Yang et al. [2023b], Hong et al. [2023], Yang et al. [2023c], utilizing a variety of tools, have demonstrated strong capabilities in task planning and reasoning.”
Tool learning with large language models: a survey (2024, unknown): “Tuning-free methods predominantly leverage the few-shot approach to provide demonstrations for parameter extraction or rule-based methods, thereby enhancing the capability of LLMs to identify parameters [93,96,126,186].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Many API-based meth-ods [73, 74, 95, 102, 120, 127, 129] have also attempted to integrate vision APIs with LLMs for solving vision-centric tasks.”
A survey on multimodal large language models (2023, unknown): “The first step is often finished by leveraging the CoT ability of LLMs. Specifically, LLMs are prompted explicitly to output task planning [80] or, more directly, the modules to call [34, 75, 76].”