AudioLM: A Language Modeling Approach to Audio Generation

By Zalán Borsos, Raphaël Marinier, Damien Vincent, Evgeny Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi and 1 more. 2022

990 citations recorded by the source corpus.

We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Dong Yu
3. Tao Qin
4. Xu Tan
5. Tie-Yan Liu
6. Yoshua Bengio
7. Mohit Bansal
8. Jianfeng Gao

In Generative Models

1. Jesse Engel
2. Yilun Du
3. Stefano Ermon
4. Wei-Ning Hsu
5. Yi Ren
6. Xu Tan
7. Tie-Yan Liu
8. Zhou Zhao

In Speech & Audio

1. Jinyu Li
2. Wei-Ning Hsu
3. Shinji Watanabe
4. Yu Zhang
5. Xu Tan
6. Yossi Adi
7. Lei He
8. Sheng Zhao

In Signal Processing

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Yu Zhang
5. Xu Tan
6. Wei-Ning Hsu
7. Yossi Adi
8. Tao Qin

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Thebaselinemodelsincludethezero-shotTTS model YourTTS [9] and speech-to-speech generation models GSLM [32] and AudioLM [33].”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, background): “Previous works [12, 13, 6] encode speech into multi-level discrete tokens using a neural audio codec [14, 15] based on residual vector quantization (RVQ).”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Discretization of speech through audio codecs [34– 36] is central to our design, as it enables the direct application of methods developed for LLMs, which underlie recent works on LTTS [16–21 , 37].”
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, methodology): “On the other hand, models like VALL-E (Wang et al., 2023a) and AudioLM (Borsos et al., 2023) utilize autoregressive Transformers to model discrete audio tokens, achieving great in-context learning performance.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “The superiority of Languge Models (LMs) in text-generative tasks inspires a series of LM-based models in audio generation (Wang et al., 2023a; Kharitonov et al., 2023; Huang et al., 2023b; Agostinelli et al., 2023; Borsos et al., 2023).”
• RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “For example, VALL-E (Wang et al., 2023a), uses them for zero-shot text to speech (TTS), and AudioLM (Borsos et al., 2023) employs these to-kens not only to produce realistic speech but also music.”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “AudioLM [4] and MusicLM [10] are pioneering efforts that exploit multiple types of tokens, each with a distinct time scale and degree of semantic granularity, allowing for hierarchical token generation.”
• AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “The subjective experiments were conducted on an earlier version of AudioPaLM using the acoustic generation method described in AudioLM [Borsos et al., 2022].”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “The literature (Borsos et al. 2022; Polyak et al. 2021) reveals that semantic tokens primarily capture articulation information while lacking sufficient acoustic details, particularly in relation to speaker identity.”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “We believe this interaction is beneficial when training hierarchical generative models on top of these codes [5, 40, 1], for example to partition the codes into “coarse” tokens (denoting the most significant codes) and “fine” tokens.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Recently, speech generation systems like AudioLM [4] and VALL-E [58] propose to utilize neural audio codec models [64, 14] to convert audio waveform into discrete codes as the intermediate representation and design LLMs to generate these codes to achieve speech synthesis.”
• PolyVoice: Language Models for Speech to Speech Translation (2023, neutral): “AudioLM (Borsos et al., 2022) is a pioneer work in using language models (LM) for audio generation.”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “While traditionally, neural vocoders reconstruct the audio waveform from a mel-scaled spectrogram – an approach widely adopted in many speech synthesis pipelines – recent research has started to utilize learnt features (Siuzdak et al., 2022), often in a quantized form (Borsos et al., 2022).”
• SoundStorm: Efficient Parallel Audio Generation (2023, positive): “In particular, AudioLM (Borsos et al., 2022) introduces a hierarchical sequence-to-sequence approach where high-level semantic tokens are generated as an intermediate representation, which is then used as a conditioning signal for predicting tokens of a SoundStream (Zeghidour et al., 2022) codec.”
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, methodology): “AudioLM [6] utilize audio codec model encode the waveform into discrete tokens, and then using Language Model (LM) to model the generation process.”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “Benefiting from these designs, NaturalSpeech 2 is more stable and robust than previous autoregressive models, and only needs one acoustic model (the diffusion model) instead of two-stage token prediction as in [21, 13], and can extend the styles beyond speech (e.”
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, neutral): “Recently, it is emerging to apply discrete audio representation learning to audio synthesis, e.g., AudioGen [Kreuk et al., 2022] and AudioLM [Borsos et al., 2022].”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “…samples are available at https://M-Voice. github.io for tasks that require the modeling of complex interactions, and they are trained to predict sequences of discrete tokens, have also been adapted to continuous, audio signals (Borsos et al., 2022; Agostinelli et al., 2023; Kreuk et al., 2022).”