Gorilla: Large Language Model Connected with Massive APIs
By Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez. 2023
1,419 citations recorded by the source corpus.
Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's code, model, data, and demo are available at https://gorilla.cs.berkeley.edu
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Cultural Studies, Large Language Models, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zhiyuan Liu
- Maosong Sun
- Mohit Bansal
- Jianfeng Gao
- Dacheng Tao
- Percy Liang
- Luke S. Zettlemoyer
- Graham Neubig
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Jie Zhou
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Bernt Schiele
- Jitendra Malik
In Cultural Studies
In Large Language Models
- Amirhossein Nadiri
- Yashar Talebirad
- Maosong Sun
- Ion Stoica
- Naman Jain
- Tianjun Zhang
- Zhiyuan Liu
- Huan Sun
In LLM Agents & Tool Use
- Amirhossein Nadiri
- Yashar Talebirad
- Maosong Sun
- Oriana Riva
- Timothy P. Lillicrap
- Jie Zhou
- Zhiyuan Liu
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “…domains and tasks such as program repair (Zheng et al., 2024; Olausson et al., 2023), optimization (Madaan et al., 2023a), test generation (Steenhoek et al., 2023), documentation generation (Luo et al., 2024), tool usage (Patil et al., 2023; Qin et al., 2024), SQL (Sun et al., 2023), and more.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “Neither of these details are important as others to mention for instruction-tuned models nor provided by the papers.”
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (2023, methodology): “When generalizing ToolLLaMA to APIBench, no training updates were made to ToolLLaMA, but instead of treating each API in the prompt as a function call.”
- ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2023, methodology): “Furthermore, in the context of tool use, several works (Schick et al. 2023; Patil et al. 2023; Yang et al. 2023) have already employed model-synthesized data to enhance specific tool-use capabilities.”
- TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, background): “These agents have shown interesting capabilities, such as tool use (Schick et al., 2023; Patil et al., 2023; Qin et al., 2024) and various forms of reasoning (Wei et al., 2022; Yao et al., 2022; Lewkowycz et al., 2022), potentially fulfilling the role of some of the cognitive substrates that were…”
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework (2023, methodology): “, a uni-direction (no back-andforth message exchange) pipeline can be also orchestrated with LangChain [24], LlamaIndex [31], Guidance [18], Semantic Kernel [43], Gorilla [40] or low-level inference API (‘autogen.”
- Mind2Web: Towards a Generalist Agent for the Web (2023, background): “The creation of recent benchmarks on tool learning [16, 22] further highlights the growing interest in evaluating LLMs’ proficiency in tool usage.”
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, neutral): “Patil et al. (2023) demonstrated single-turn function calling with narrow functions limited to specific APIs, and more recent work has expanded the scope and complexity of these systems (e.g., Wang et al. (2024)).”
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “…finetuning – in common-sense reasoning (West et al., 2022), text-summarization (Sclar et al., 2022), mathematical reasoning (Luo et al., 2023a; Yue et al., 2023), tool use (Patil et al., 2023), coding (Luo et al., 2023b), and general-purpose reasoning Li et al. (2023b); Zelikman et al. (2022).”
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “( 2) Fine-tuned Method : This approach involves the fine-tuning of LLMs using function calling examples (Schick et al., 2023; Patil et al., 2023; Qin et al., 2023; OpenAI, 2023b).”
- Android in the Wild: A Large-Scale Dataset for Android Device Control (2023, background): “Finally, LLMs have shown impressive results leveraging APIs, when they are available, for performing higher-level tasks [36, 33, 34].”
- Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions (2023, background): “More recently, generative pre-trained foundation models have demonstrated impressive in-context learning abilities, allowing them to tackle decision-making tasks that require logic reasoning [8] and/or interactions with external APIs [9, 13, 22].”
- CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024, background): “Examples such as ToolFormer (Schick et al., 2023), HuggingGPT (Shen et al., 2023), Gorilla (Patil et al., 2023), ToolLlama (Qin et al., 2023) and ToolCoder (Zhang et al., 2023a) demonstrate LLMs’ proficiency in tool utilization.”
- PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “The first category includes models that employ a large language model to interface with individual, modality-specific models or APIs [14, 17, 35, 42, 47].”
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, background): “LLMs can be connected with real-world Application Programming Interfaces (APIs), enabling them to actively engage with diverse external tools (Qin et al., 2023b; Parisi et al., 2022; Schick et al., 2023a; Tang et al., 2023; Patil et al., 2023; Song et al., 2023; Hao et al., 2023; Yuan et al., 2023).”
- Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents (2023, positive): “The Gorilla system [12] is based on a fine-tuned LLaMA [13] model with additional capabilities to retrieve documents and integrate this information during both training and inference.”
- ToolQA: A Dataset for LLM Question Answering with External Tools (2023, methodology): “…issues by selecting data from out-of-scope sources that have not been memorized by LLMs. Concurrent with our work, there are several recent benchmarks for evaluating LLMs’ ability in using multiple tools for solving challenging tasks, including API-Bank [29], APIBench [48], and ToolBench [52, 71].”
- Visual Adversarial Examples Jailbreak Aligned Large Language Models (2023, background): “, robotics [37; 22; 9], APIs management [60], making tools [11], developing plug-ins [43], the implications of our attacks may further expand according to specific downstream applications.”
- RAFT: Adapting Language Model to Domain Specific RAG (2024, methodology): “…Supervised-finetuning both with-and without-RAG across PubMed (Dernoncourt & Lee, 2017), HotpotQA (Yang et al., 2018), and HuggingFace Hub, Torch Hub, and Tensorflow Hub Gorilla datasets (Patil et al., 2023), presenting a novel, yet simple technique to improve pre-trained LLMs for in-domain RAG.”