AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

By Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jia-Bin Huang, Jinglin Liu and 3 more. 2023

405 citations recorded by the source corpus.

Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. Despite the recent success, current LLMs are not capable of processing complex audio information or conducting spoken conversations (like Siri or Alexa). In this work, we propose a multi-modal AI system named AudioGPT, which complements LLMs (i.e., ChatGPT) with 1) foundation models to process complex audio information and solve numerous understanding and generation tasks; and 2) the input/output interface (ASR, TTS) to support spoken dialogue. With an increasing demand to evaluate multi-modal LLMs of human intention understanding and cooperation with foundation models, we outline the principles and processes and test AudioGPT in terms of consistency, capability, and robustness. Experimental results demonstrate the capabilities of AudioGPT in solving 16 AI tasks with speech, music, sound, and talking head understanding and generation in multi-round dialogues, which empower humans to create rich and diverse audio content with unprecedented ease. Code can be found in https://github.com/AIGC-Audio/AudioGPT

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, background): “Recently, many works [1, 2, 5, 7, 8, 17, 18] propose to model speech and audio in the discrete latent space with the help of Audio Codec.”
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding (2023, background): “Multimodal Instruction Tuning Recently, instruction tuning has been expanded to the multi-modal setting, including image, video [24, 25], and audio [26, 27].”
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, background): “…task or dimension, like task completion (Liu et al., 2023c), code generation (Yang et al., 2023a), human-LLM collaborative task solving (Lee et al., 2022b; Huang et al., 2023b; Fu et al., 2023), tool manipulation (Tang et al., 2023), and web nevigation (Zhou et al., 2023; Deng et al., 2023a).”
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, background): “Large language models (LLMs) have taken the world by storm, showing their ability to generate high-quality text for various tasks like storytelling, serving as chat assistants, and even composing music [1, 11].”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, methodology): “For the audio modality, there have been attempts to utilize well-trained audio foundation models as tools, such as AudioGPT (Huang et al., 2023) and HuggingGPT (Shen et al., 2023), while leveraging LLMs as a versatile interface.”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, background): “…from unified audio-text modeling, some studies integrate expert audio models with LLMs to enable direct audio interaction with LLMs, such as HuggingGPT (Shen et al., 2023) and AudioGPT (Huang et al., 2023b); however these models have high complexity , resource consumption , and error accumulation .”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “…(Wang et al., 2024c), MLLM-Tool(Wang et al., 2024a) Design Division Tool-using Visual ChatGPT(Wu et al., 2023a), ViperGPT(Surís et al., 2023), MM-REACT(Yang et al., 2023b), HuggingGPT (Shen et al., 2023),AudioGPT (Huang et al., 2023b), ControlLLM (Liu et al., 2023i), LLaVA-Plus (Liu et al.)”
The rise and potential of large language model based agents: a survey (2025, positive): “Apart from the cascading paradigm [218, 219, 269], audio information encoding can also enhance the seamless integration of audio with other modalities of inputs [222].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “This includes video inputs [110, 126], audio [80] or both [216].”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “To achieve LLMs with both multimodal input and output, some thus explore employing LLMs as decision-makers, and utilizing existing off-the-shelf multimodal encoders and decoders as tools to execute multimodal input and output, such as Visual-ChatGPT [88], HuggingGPT [72], and AudioGPT [34].”
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023, methodology): “The current speech-language model mainly adopts a cascading paradigm (Huang et al., 2023a) i.e., the LLM is connected with an automatic speech recognition (ASR) model or a text-to-speech (TTS) model in tandem, or the LLM is employed as a control hub, with several speech processing models (Cheng et…”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “A new horizon emerges as researchers begin to explore multi-modal LLMs, capable of processing various forms of data such as audio [17] and images [1, 18, 24, 27, 32, 54, 56].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023, methodology): “…Silent Video Audio BLIP2 (Li et al., 2023b) MiniGPT4 (Zhu et al., 2023) LLaVA (Liu et al., 2023) mPLUG-Owl (Ye et al., 2023) VideoChat (Li et al., 2023c) AudioGPT (Huang et al., 2023a) Video-ChatGPT (Maaz et al., 2023) Video-LLaMA Table 1: Comparison with popular multi-modal large language models.”