SoundStream: An End-to-End Neural Audio Codec
By Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi. IEEE/ACM Transactions on Audio Speech and Language Processing, 2021
1,407 citations recorded by the source corpus.
We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to quantizer layers, a single model can operate across variable bitrates from 3 kbps to 18 kbps, with a negligible quality loss when compared with models trained at fixed bitrates. In addition, the model is amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU. In subjective evaluations using audio at 24 kHz sampling rate, SoundStream at 3 kbps outperforms Opus at 12 kbps and approaches EVS at 9.6 kbps. Moreover, we are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency, which we demonstrate through background noise suppression for speech.
Subfields: Artificial Intelligence, Speech & Audio, Signal Processing
View Paper: https://arxiv.org/abs/2107.03312
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Speech & Audio
In Signal Processing
Citations in the corpus
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “AudioLM [33] utilizes a similar approach but employs audio codecs [36] to synthesize speech in tandem with semantic codes.”
- High Fidelity Neural Audio Compression (2022, methodology): “For fair evaluation, we also compare EnCodec to our reimplementation of SoundStream (Zeghidour et al., 2021).”
- VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “These works typically employ SoundStream [Zeghidour et al., 2021] and Encodec [Défossez et al., 2022], initially designed for speech compression, as the neural codec model.”
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “We compare the proposed FACodec in terms of the reconstruction quality with strong baselines, such as EnCodec [15], HiFi-Codec [65], Descript-Audio-Codec (DAC) [32], and our reproduced SoundStream [14].”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “After the writer the editor the publisher hired fired quit, the company found itself in quite a bind.”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “Given any hidden output h t , the integer vector z t = [ z 1 t , ..., z n q t ] is generated by Residual Vector Quantization (RVQ) (Zeghidour et al., 2021), where n q denotes the number of vector quantization layers.”
- FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “This is followed by a residual vector quantizer (RVQ) [3, 4] to obtain parallel token streams.”
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “When M = 1 , RVQ is equivalent to VQ.”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Large scale in-context learning models: With the advancement in neural codec for speech [22, 12, 67], many recent studies explore token-based language modeling for speech generation.”
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “, 2021], which a second stage then uses as conditioning to generate the “acoustic” tokens of a SoundStream [Zeghidour et al., 2021] neural codec.”
- UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, background): “To be or not to be, that is the question. models like Soundstream (Zeghidour et al. 2022) and En-codec (D´efossez et al. 2022), are trained specifically for speech reconstruction.”
- High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “We now compare the performance of our final model with competitive baselines: EnCodec [8], Lyra [45], and Opus [36], a popular open-source audio codec.”
- Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Instead of using latent encoded by audio codec as the intermediate representation [64, 14, 58], we decompose mel-spectrogram into content, timbre, prosody, and phase attributes and model each of them according to their intrinsic properties.”
- PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “After the U2S model predict the acoustic tokens represented by the SoundStream codec, the decoder of SoundStream reconstruct them to the waveform.”
- Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “However, we adopt a hinge loss formulation instead of the least squares GAN objective, as suggested by Zeghidour et al. (2021): where D k is the k th subdiscriminator.”
- Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, methodology): “Since AudioDec adopts an E2E AE-based architecture with the SoundStram backbone [25], the E2E AE-based audio codec foundations and the baseline SoundStream are introduced in this chapter.”
- SoundStorm: Efficient Parallel Audio Generation (2023, positive): “In particular, AudioLM (Borsos et al., 2022) introduces a hierarchical sequence-to-sequence approach where high-level semantic tokens are generated as an intermediate representation, which is then used as a conditioning signal for predicting tokens of a SoundStream (Zeghidour et al., 2022) codec.”
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “The training of the neural codec follows the loss function in [19].”