High Fidelity Neural Audio Compression

By Alexandre D'efossez, Jade Copet, Gabriel Synnaeve, Yossi Adi. Trans. Mach. Learn. Res., 2022

1,262 citations recorded by the source corpus.

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

Subfields: Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

View Paper: https://arxiv.org/abs/2210.13438

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “We adopt a pre-trained neural audio codec model, EnCodec [Défossez et al., 2022], as our tokenizer.”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “We use Byte-Pair Encoding (BPE) for text tokenization, and the pre-trained open-sourced EnCodec model [Défossez et al., 2022] at 6K bitrates for 24kHz audio reconstruction for speech tokenization.”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Previous works [12, 13, 6] encode speech into multi-level discrete tokens using a neural audio codec [14, 15] based on residual vector quantization (RVQ).”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “After the writer the editor the publisher hired fired quit, the company found itself in quite a bind.”
Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “The watermarking model has similar building block as Encodec Défossez et al. (2022).”
UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “Following Wang et al. (2023a); Kharitonov et al. (2023), neural codec models (Défossez et al., 2022; Yang et al., 2023b) are used in this work for audio tokenization.”
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “We find that this hyperparameter setting is suitable for speech signals, and the loss balancer [6] is also a work in progress.”
RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “Although EnCodec (Défossez et al., 2022) preserves information beyond linguistic content of the speech, its performance is inferior to RepCodec in terms of semantic information.”
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “The acoustic prompt was obtained by applying EnCodec to the noisy signal.”
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, background): “It is worth noting that MOS scores for naturalness and speaker similarity achieved by UniCATS are even higher 6 https://github.com/lifeiteng/vall-e than those of Encodec resynthesis, indicating that our model breaks the upper bound of a series of other works that uses acoustic tokens.”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “Our decoder has 4 corresponding layers, which upsample at rates [8, 8, 4, 2].”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “These systems utilize the neural audio codec models [64, 14] to convert audio waveform into latent and consider it as the intermediate representation for speech generation.”
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “, 2023) model transforms continuous speech signal to discrete codec codes with EnCodec model (Défossez et al., 2022), and trains a neural codec language model by scaling up TTS training data to 60K hours.”
SoundStorm: Efficient Parallel Audio Generation (2023, positive): “Neural audio codecs (Zeghi-dour et al., 2022; D´efossez et al., 2022), with their ability of reconstructing high-quality audio at very low bitrates, subsequently allowed for extending discrete modeling to audio signals as diverse as multi-speaker speech and piano (Bor-sos et al., 2022; Kharitonov…”
HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, methodology): “We follow the same encoder and decoder design from Encodec [7] and SoundStream [8], which is a convolutional-based encoder-decoder architecture with sequential modeling component applied over the latent representation, both on the encoder and on the decoder side.”
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “2) There is a dilemma between the codec and language model: on the one hand, the codec with token quantization (VQ-VAE [16, 17] or VQ-GAN [18]) usually has a low bitrate token sequence, which, although eases the language model generation, incurs information loss on the high-frequency fine-grained ac”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “We also adopt the neural codec model EnCodec [Défossez et al., 2022] as the acoustic quantizer, which is an encoder-decoder model with L quantization layers.”