SoundStorm: Efficient Parallel Audio Generation

By Zalán Borsos, Matthew Sharifi, Damien Vincent, Evgeny Kharitonov, Neil Zeghidour, Marco Tagliasacchi. 2023

162 citations recorded by the source corpus.

We present SoundStorm, a model for efficient, non-autoregressive audio generation. SoundStorm receives as input the semantic tokens of AudioLM, and relies on bidirectional attention and confidence-based parallel decoding to generate the tokens of a neural audio codec. Compared to the autoregressive generation approach of AudioLM, our model produces audio of the same quality and with higher consistency in voice and acoustic conditions, while being two orders of magnitude faster. SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4. We demonstrate the ability of our model to scale audio generation to longer sequences by synthesizing high-quality, natural dialogue segments, given a transcript annotated with speaker turns and a short prompt with the speakers' voices.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “For example, Soundstorm [51] leveraged the conﬁdence-based parallel decoding scheme [52] to generate the acoustic codec codes with a non-autoregressive model.”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “To improve modeling efficiency, some works explore fully non-autoregressive methods for zero-shot TTS [Borsos et al., 2023, Le et al., 2024, Ju et al., 2024].”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…these systems into four groups based on data representation and modelling methods: 1) Discrete Tokens + Autoregressive [6, 19, 20]; 2) Discrete Tokens + Non-2 autoregressive [13, 21, 22]; 3) Continuous Vectors + Autoregressive [23]; 4) Continuous Vectors + Non-autoregressive [5, 11, 24, 25].”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Some of these models, described in depth in Section 5, rely on causal language modeling tasks, like AudioLM [16] or VALL-E [17], whereas others use non-causal modules, such as SoundStorm [18] or SpeechX [19], or diffusion decoders [20, 21].”
Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “…models can mimic any vocal style using audio prompts that are as short as three seconds (Wang et al., 2023a; Shen et al., 2023; Le et al., 2023; Kharitonov et al., 2023), infill a portion of speech to remove transient noise or edit words for any speaker (Le et al., 2023; Shen et al., 2023),…”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “We rerun this using a SoundStorm model instead, and measure the impact on the CVSS S2ST task.”
Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, methodology): “This approach is already used in SOTA models such as VALL-E (Wang et al., 2023), SPEAR-TTS (Kharitonov et al., 2023) and SoundStorm (Borsos et al., 2023b).”
HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “Recently, VoiceBox [42] and P-Flow [43] used conditional ﬂow matching (CFM) with optimal transport for fast sampling.”