Pengi: An Audio Language Model for Audio Tasks

By Soham Deshmukh, Benjamin Elizalde, Rita Singh, Huaming Wang. Neural Information Processing Systems, 2023

321 citations recorded by the source corpus.

In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question&Answering. We introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 22 downstream tasks, our approach yields state-of-the-art performance in several of them. Our results show that connecting language models with audio models is a major step towards general-purpose audio understanding

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ad22af138fa1d1490cda0301abf8159a7c30c5a2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiliang Zhang
2. Yichong Leng
3. Zhihao Du
4. Jingren Zhou
5. Ziyang Ma
6. Xiaohuan Zhou
7. Zhijie Yan
8. Kai Hu

In Signal Processing

1. Yichong Leng
2. Zhihao Du
3. Jingren Zhou
4. Shiliang Zhang
5. Ziyang Ma
6. Zhijie Yan
7. Kai Hu
8. Siqi Zheng

Citations in the corpus

• Qwen2.5-Omni Technical Report (2025, unknown): “CLAP (Elizalde et al., 2022) 0.495 Pengi (Deshmukh et al., 2023) 0.604 Qwen-Audio (Chu et al., 2023a) 0.929 Qwen2-Audio (Chu et al., 2024a) 0.939 Qwen2.”
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models (2023, background): “Existing representative audio-language multi-tasks language models, such as SpeechNet (Chen et al., 2021), SpeechT5 (Ao et al., 2021), VIOLA (Wang et al., 2023d), Whisper (Radford et al., 2023) and Pengi (Deshmukh et al., 2023) are limited to processing specific audio types, such as human speech or ”
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, background): “The third group, such as Whisper (Radford et al., 2022), USM (Zhang et al., 2023b), and Pengi (Deshmukh et al., 2023), focus on converting continuous speech features into text, but they cannot support audio generation tasks .”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
• A survey on multimodal large language models (2023, unknown): “We write this survey to provide researchers with a grasp of the basic idea, main method, and current progress of MLLMs. Note that we mainly focus on visual and language modalities, but also include works involving other modalities.”
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, unknown): “This categorization ensures a structured evaluation of model performance across varying levels of complexity.”
• Qwen2-Audio Technical Report (2024, methodology): “…Performance of Qwen2-Audio, Qwen-Audio and previous top-tiers from LALMs such as SpeechT5 (Ao et al., 2021), SpeechNet (Chen et al., 2021), SpeechLLaMA (Wu et al., 2023a), SALMONN (Tang et al., 2024), Whisper (Radford et al., 2023) Pengi (Deshmukh et al., 2023), and SpeechVerse (Das et al., 2024).”