LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT

By Jiaming Wang, Zhihao Du, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma and 5 more. 2023

115 citations recorded by the source corpus.

Generative Pre-trained Transformer (GPT) models have achieved remarkable performance on various natural language processing tasks, and have shown great potential as backbones for audio-and-text large language models (LLMs). Previous mainstream audio-and-text LLMs use discrete audio tokens to represent both input and output audio; however, they suffer from performance degradation on tasks such as automatic speech recognition, speech-to-text translation, and speech enhancement over models using continuous speech features. In this paper, we propose LauraGPT, a novel unified audio-and-text GPT-based LLM for audio recognition, understanding, and generation. LauraGPT is a versatile LLM that can process both audio and text inputs and generate outputs in either modalities. We propose a novel data representation that combines continuous and discrete features for audio: LauraGPT encodes input audio into continuous representations using an audio encoder and generates output audio from discrete codec codes. We propose a one-step codec vocoder to overcome the prediction challenge caused by the multimodal distribution of codec tokens. We fine-tune LauraGPT using supervised multi-task learning. Extensive experiments show that LauraGPT consistently achieves comparable to superior performance compared to strong baselines on a wide range of audio tasks related to content, semantics, paralinguistics, and audio-signal analysis, such as automatic speech recognition, speech-to-text translation, text-to-speech synthesis, speech enhancement, automated audio captioning, speech emotion recognition, and spoken language understanding.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ziyang Ma
2. Zhuo Chen
3. Xie Chen
4. Ya-Zhen Song
5. Xiaofei Wang
6. Dong Yu
7. Jinyu Li
8. Yi Luo

In Signal Processing

1. Zhuo Chen
2. Xie Chen
3. Ziyang Ma
4. Ya-Zhen Song
5. Xiaofei Wang
6. Dong Yu
7. Jinyu Li
8. Yi Luo

Citations in the corpus

• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “In light of this, VALL-E (Wang et al., 2023a) and subsequent works (Kharitonov et al., 2023; Rubenstein et al., 2023; Wang et al., 2023b) have successfully employed decoder-only language model for zero-shot TTS.”