SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

By Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, Xipeng Qiu. 2023

694 citations recorded by the source corpus.

Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-model content. With discrete speech representations, we first construct SpeechInstruct, a large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow multi-modal human instructions and highlight the potential of handling multiple modalities with one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wayne Xin Zhao
2. Ji-rong Wen
3. Yupeng Hou
4. Shinji Watanabe
5. Hongyu Lu
6. Lidong Bing
7. Xin Li
8. Bowen Zheng

Citations in the corpus

• RepCodec: A Speech Representation Codec for Speech Tokenization (2023, background): “Acoustic tokens are produced by audio codecs (Zeghidour et al., 2022; Défossez et al., 2022; Zhang et al., 2023b), which aim to reconstruct the original audio so that it can be perceptually identical to listeners.”
• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “In parallel with our work, SpeechGPT (Zhang et al., 2023a) was recently proposed to perceive and generate multi-model content.”
• SVIT: Scaling up Visual Instruction Tuning (2023, background): “There are also some works that contribute multimodal instruction data of videos (Li et al., 2023b), RGB-D images (Li et al., 2023b), speech (Zhang et al., 2023a), audio (Zhang et al., 2023b), etc.”
• PandaGPT: One Model To Instruction-Follow Them All (2023, background): “…been made in text-to-image retrieval and generation [18], visually-grounded instruction following [12, 31], and speech understanding and generation [29], these advances have largely been confined to separate combinations of text and other modalities or, at best, a few visual modalities (e.g.,…”
• Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023, background): “In order to further explore the potential of LLMs, many researchers attempt to endow LLMs with the capability of understanding multi-modal content (Huang et al., 2023a; Zhang et al., 2023b; Yin et al., 2023).”
• A survey on multimodal large language models (2023, neutral): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, methodology): “SpeechGPT (Zhang et al., 2023a) first converts human speech into discrete HuBERT tokens (Hsu et al., 2021), and then designs a three-stage training pipeline on paired speech data, speech instruction data and chain-of-modality instruction data accordingly.”
• SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, methodology): “When LLM-based speech synthesis is also considered, the LLM output space can be augmented with speech tokens as well, such as SpeechGPT (Zhang et al., 2023a) and AudioPaLM (Rubenstein et al., 2023).”
• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, background): “AI-basedaudiogenerationhas a wide potential in applications including synthesizing human or artificial voices for digital assistants [5], generating sound effects and background music for movies, and games [6], and automating the production of podcasts and audiobooks [7].”
• Qwen2-Audio Technical Report (2024, neutral): “Lastly, to objectively evaluate the chat capabilities of Qwen2-Audio, we measured its performance on the (Deshmukh et al., 2023) 0.6035 Qwen-Audio (Chu et al., 2023) 0.9289 Qwen2-Audio 0.9392 AIR-Bench (Yangetal., 2024) (Tang et al., 2024) BLSP (Wang et al., 2023a) Pandagpt (Su et al., 2023) Macaw-L”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
• VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, methodology): “They also require additional losses like an alignment loss to encourage cross-modal transfer between speech [18–20], which are similar to our approach.”
• Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2023, background): “It can be an analogy with the mutual association or mapping between images and text [29], or the conversion between speech and text [45].”
• NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “…similar practices for building MM-LLMs that are able to understand videos (e.g., Video-Chat [44] and Video-LLaMA [103]), audios (e.g., SpeechGPT [102]), etc. Profoundly, PandaGPT [77] achieves a comprehensive understanding of six different modalities simultaneously by integrating the multimodal…”
• MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “However, both Q, P, MQ-Former require an additional PT process for initialization.”
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, neutral): “This results in high latency, negatively impacting user experience in applications like voice chat [36,37].”
• SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models (2023, background): “Spoken Generative Language Models Speech discrete representation based spoken generative language models have demonstrated remarkable performance on various speech processing tasks (Borsos et al., 2022; Wang et al., 2023; Kharitonov et al., 2023; Zhang et al., 2023).”