Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis

By Hubert Siuzdak. 2023

266 citations recorded by the source corpus.

Recent advancements in neural vocoding are predominantly driven by Generative Adversarial Networks (GANs) operating in the time-domain. While effective, this approach neglects the inductive bias offered by time-frequency representations, resulting in reduntant and computionally-intensive upsampling operations. Fourier-based time-frequency representation is an appealing alternative, aligning more accurately with human auditory perception, and benefitting from well-established fast algorithms for its computation. Nevertheless, direct reconstruction of complex-valued spectrograms has been historically problematic, primarily due to phase recovery issues. This study seeks to close this gap by presenting Vocos, a new model that directly generates Fourier spectral coefficients. Vocos not only matches the state-of-the-art in audio quality, as demonstrated in our evaluations, but it also substantially improves computational efficiency, achieving an order of magnitude increase in speed compared to prevailing time-domain neural vocoding approaches. The source code and model weights have been open-sourced at https://github.com/gemelo-ai/vocos.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Shujie Liu
4. Lei He
5. Sheng Zhao
6. Zhuo Chen
7. Tao Qin
8. Yu Wu

In Statistical and Nonlinear Physics

1. Zhuo Chen
2. Jinyu Li
3. Naoyuki Kanda
4. Sanyuan Chen
5. Shujie Liu
6. Yanqing Liu
7. Ziyang Ma
8. Long Zhou

In Generative Models

1. Zhuo Chen
2. Jinyu Li
3. Sanyuan Chen
4. Shujie Liu
5. Yanqing Liu
6. Ziyang Ma
7. Long Zhou
8. Zhikang Niu

In Speech & Audio

1. Lei He
2. Xu Tan
3. Sheng Zhao
4. Yanqing Liu
5. Jinyu Li
6. Shujie Liu
7. Zhuo Chen
8. Sanyuan Chen

In Signal Processing

1. Jinyu Li
2. Lei He
3. Xu Tan
4. Shujie Liu
5. Zhuo Chen
6. Yu Wu
7. Sheng Zhao
8. Sanyuan Chen

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Additionally, the open-sourced pre-trained Vocos model [62] serves as an alternative audio codec decoder for speech generation.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “These include Vocos [Siuzdak, 2023], SpeechTokenizer [Zhang et al., 2023a], AudioDec [Wu et al., 2023], AcademiCodec [Yang et al., 2023a], Descript-audio-codec (DAC) [Kumar et al., 2024], FunCodec [Du et al., 2024b], and RepCodec [Huang et al., 2023b].”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “…work by expanding the tasks supported, enhancing robustness and quality (e.g., stabilizing the AR generation with duration control [41] and leveraging improved neural codecs [42]), and developing more advanced conditioning mechanisms (e.g., us-ing disentangled prosody and speaker information [43]).”
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “Its predecessor ConvNeXt V1 [49] is used in many works and shows a strong temporal modeling capability in speech domain tasks [50, 51].”
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models (2024, unknown): “Codec language models utilize a speech codec model to extract discrete speech representation [9–11] and employ an autoregressive [8, 12–17] or masked [18] language model to predict the speech tokens, which are then synthesized to waveforms via codec vocoders [19,20].”