AudioPaLM: A Large Language Model That Can Speak and Listen

By Paul K. Rubenstein, Chulayuth Asawaroengchai, D. Nguyen, Ankur Bapna, Zalán Borsos, Felix de Chaumont Quitry, Peter Chen, D. Badawy, Wei Han, Evgeny Kharitonov and 21 more. 2023

486 citations recorded by the source corpus.

We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples

Subfields: Artificial Intelligence, Large Language Models, Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Xu Tan
5. Tao Qin
6. Tie-Yan Liu
7. Shujie Liu
8. Tara N. Sainath

In Large Language Models

1. Furu Wei
2. Rohit Prabhavalkar
3. Jiang Bian
4. Ralf Schlüter
5. David A. Ross
6. Alex Wang
7. Takaaki Hori
8. Jinyu Li

In Multimodal Learning

1. Jinyu Li
2. José Lezama
3. Alex Wang
4. Zhuo Chen
5. Hassan Akbari
6. Ming-Hsuan Yang
7. Kihyuk Sohn
8. Shiliang Zhang

In Speech & Audio

1. Jinyu Li
2. Shinji Watanabe
3. Xu Tan
4. Sheng Zhao
5. Tara N. Sainath
6. Rongjie Huang
7. Shujie Liu
8. Zhuo Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “AudioPaLM [73] further integrated the codec tokens into the LLM PaLM 2 [74], demonstrating promising results on speech recognition and translation tasks.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “AudioPaLM [Rubenstein et al., 2023] fused the codec tokens into the LLM PaLM 2 [Anil et al., 2023], and demonstrated promising results on speech recognition and translation tasks.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “These continuous representations are highly informative and can be adopted in both speech understanding (Rubenstein et al., 2023) and generative tasks (Borsos et al., 2023).”
• FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, background): “Moreover, neural speech codecs facilitate the modeling of speech and text within a single framework, enabling the model to both listen and speak [17,18].”
• RepCodec: A Speech Representation Codec for Speech Tokenization (2023, background): “Acoustic tokens are produced by audio codecs (Zeghidour et al., 2022; Défossez et al., 2022; Zhang et al., 2023b), which aim to reconstruct the original audio so that it can be perceptually identical to listeners.”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “Notably, generative models using multi-modal input have emerged as a remarkable innovation [5], [6], [7], [8], [9], [10].”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “Rubenstein et al. (2023) fuse text-based and speech-based language models into a unified multitask architecture to process and generate text and speech.”
• VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, background): “While Large Language Models (LLMs) are commonly used as foundational models across various modalities including language [3, 8], code [38, 43], audio [51], speech [1], and robotics [20, 86], the diffusion model remains the predominant approach for video generation.”
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, unknown): “Currently, advanced Large Audio-Language Models (LALMs) are primarily evaluated on foundational tasks such as Automatic Speech Recognition (ASR), Acoustic Scene Classification, or Music Genre Classification (Rubenstein et al., 2023; Gong et al., 2023c; Ghosh et al., 2024c).”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “Numerous audio-text M-LLMs, leveraging transformer-based architectures, have integrated text-and audio-based language models, such as AudioPaLM [97], AudioLM [98], Pengi [99], AudioGPT [100], SpeechGPT [49], VioLA [101], and SALMONN [102], into a unified multimodal architecture.”
• MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, unknown): “However, both Q, P, MQ-Former require an additional PT process for initialization.”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, background): “Besides, recent works [22, 30, 33] have shown the decoder-only architecture can be used to learn multiple tasks, as the input and output are processed jointly by a language model, and the model can be signaled to generate results for different tasks by inputting pre-defined special tokens.”
• A survey on multimodal large language models (2023, unknown): “In order to further inspect what roles LLMs exactly play in LLM-Aided Visual Reasoning systems, existing related works are divided into three types: • LLM as a Controller • LLM as a Decision Maker”
• SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, methodology): “When LLM-based speech synthesis is also considered, the LLM output space can be augmented with speech tokens as well, such as SpeechGPT (Zhang et al., 2023a) and AudioPaLM (Rubenstein et al., 2023).”
• End-to-End Speech Recognition: A Survey (2023, background): “In addition, in line with the growing interest in large language models in recent years, research has also begun on solving multiple tasks, including speech recognition, using only an auto-regressive, GPT-style decoder [110], [111].”
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, background): “…their capabilities to generate content in various modalities, asserting their dominance in other domains like audio (Agostinelli et al., 2023), speech (Rubenstein et al., 2023), code generation (Li et al., 2023), medical applications (Singhal et al., 2023) and robotics (Zitkovich et al., 2023).”