HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

By Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Y. Zhuang. Neural Information Processing Systems, 2023

1,596 citations recorded by the source corpus.

Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence. While there are numerous AI models available for various domains and modalities, they cannot handle complicated AI tasks autonomously. Considering large language models (LLMs) have exhibited exceptional abilities in language understanding, generation, interaction, and reasoning, we advocate that LLMs could act as a controller to manage existing AI models to solve complicated AI tasks, with language serving as a generic interface to empower this. Based on this philosophy, we present HuggingGPT, an LLM-powered agent that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according to their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according to the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT can tackle a wide range of sophisticated AI tasks spanning different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves a new way towards the realization of artificial general intelligence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/d1120d67b700e4dfe8b39eb1e48fbdea4e1a0c43

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, methodology): “Some of these methods focus on images, such as VisualChat-GPT [46] and HuggingGPT [40], while MM-REACT [48] and ViperGPT [42] can also handle videos.”
MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, methodology): “These approaches [48, 39, 11, 57, 57, 52] take an LLM as a dispatch scheduler and connect different expert models through it to allow for different tasks.”
A survey on large language model based autonomous agents (2023, neutral): “In the following, we introduce each of them more in detail.”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “These methods are exemplified by VisualChatGPT [60], HuggingGPT [52], MM-REACT [65], and ViperGPT [54].”
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023, background): “…with access to external tools and resources, such as the information retriever (Nakano et al., 2021; Shi et al., 2023; Lazaridou et al., 2022), calculator (Cobbe et al., 2021), code interpreter (Wang et al., 2022), planner (Liu et al., 2023a), and other pre-trained models (Shen et al., 2023).”
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, neutral): “Impact of LLM’s capability.”
Tool learning with large language models: a survey (2024, neutral): “Additionally, the integration of advanced natural language processing tools can augment the linguistic understanding of LLMs, thereby optimizing dialogue management and intent recognition [18, 89, 90].”
Executable Code Actions Elicit Better LLM Agents (2024, background): “…a variety of applications and comes with unique advantages: (1) Integrated with a Python interpreter, CodeAct can execute code actions and dynamically adjust prior actions or emit new action based on observations (e.g., code execution results) it receives through multiple turns of interactions.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, background): “Moving beyond merely integrating LLMs into applications, there are ongoing active research and engineering efforts to create more autonomous systems [40, 56, 72] (i.”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “Several additional works [56, 32, 57] expand the scope of tool utilization by incorporating different types of basic tools ( e.g. calculator, calendar, machine translation) to solve complex reasoning tasks.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, neutral): “Works in this category include VisualChatGPT [35], HuggingGPT [29], Cola [8], X-GPT [42], MM-REACT [37], and ViperGPT [31].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Pre-training allows the models to align different modalities and enhance the representation learning ability of the model encoder.”
Large Language Models as Tool Makers (2023, background): “Various studies (Liu et al., 2023; Parisi et al., 2022; Schick et al., 2023; Shen et al., 2023; Lu et al., 2023; Paranjape et al., 2023; Liang et al., 2023) have demonstrated that supplementing LLMs with tools, such as calculators, search engines, translation systems, calendars, or even API calls…”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024, background): “A common analogy for AI-powered applications is that the LLM acts as an operating system: it executes the instructions, determines the control flow, and handles data storage (Weng, 2023; Shen et al., 2024).”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, methodology): “Initially, MLLMs were treated as dispatch schedulers to connect vision expert models, such as VisualChatGPT [63], HuggingGPT [51], and MM-REACT [68], in order to extend language models to other tasks and modalities.”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, methodology): “For the audio modality, there have been attempts to utilize well-trained audio foundation models as tools, such as AudioGPT (Huang et al., 2023) and HuggingGPT (Shen et al., 2023), while leveraging LLMs as a versatile interface.”