BigVGAN: A Universal Neural Vocoder with Large-Scale Training

By Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, Sung-Hoon Yoon. 2022

483 citations recorded by the source corpus.

Despite recent progress in generative adversarial network (GAN)-based vocoders, where the model generates raw waveform conditioned on acoustic features, it is challenging to synthesize high-fidelity audio for numerous speakers across various recording environments. In this work, we present BigVGAN, a universal vocoder that generalizes well for various out-of-distribution scenarios without fine-tuning. We introduce periodic activation function and anti-aliased representation into the GAN generator, which brings the desired inductive bias for audio synthesis and significantly improves audio quality. In addition, we train our GAN vocoder at the largest scale up to 112M parameters, which is unprecedented in the literature. We identify and address the failure modes in large-scale GAN training for audio, while maintaining high-fidelity output without over-regularization. Our BigVGAN, trained only on clean speech (LibriTTS), achieves the state-of-the-art performance for various zero-shot (out-of-distribution) conditions, including unseen speakers, languages, recording environments, singing voices, music, and instrumental audio. We release our code and model at: https://github.com/NVIDIA/BigVGAN

Subfields: Biomedical Engineering, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Dong Yu
2. Nima Mesgarani
3. Rongjie Huang
4. Cong Han
5. Xu Tan

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Rongjie Huang
4. Sheng Zhao
5. Zhou Zhao
6. Dong Yu
7. Nima Mesgarani
8. Dongchao Yang

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Dong Yu
4. Sheng Zhao
5. Nima Mesgarani
6. Zhou Zhao
7. Rongjie Huang
8. Cong Han

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “The basic architecture of FACodec encoder and decoder follows [32] and employs the SnakeBeta activation function [68].”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “The output of a HiFi-GAN based decoder block [53] is fed into BigVGAN vocoder [54] to predict the waveform.”
• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, background): “However, HiFi-GAN does not generalize well to non-speech audio such as sound or music (Lee et al., 2022).”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “However, their efficiency is limited compared to non-iterative methods, like GAN-based models [30, 31, 32], due to the need to iteratively sample mel-spectrograms, waveforms, or other latent representations proportional to the target speech duration [15].”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “To add a periodic inductive bias to the generator, we adopt the Snake activation function proposed by Liu et al. [47] and introduced to the audio domain in the BigVGAN neural vocoding model [21].”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “We train the enhanced vocoder with the weighted sum of the least-square adversarial loss, the feature matching loss, and the spectral regression loss on mel-spectrogram, where the training objective formulation and hyperparam-eters follow Kong et al. (2020); Lee et al. (2022).”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “…the synthesizer S 3 , we train the enhanced vocoder with the weighted sum of the least-square adversarial loss, the feature matching loss, and the spectral regression loss on mel-spectrogram, where the training objective formulation and hyperparameters follow Kong et al. [20], Lee et al. [22].”
• Improve GAN-based Neural Vocoder using Truncated Pointwise Relativistic Least Square GAN (2022, methodology): “HiFiGAN[13] is a typical neural vocoder that can generate high-fidelity waveforms in real-time on a normal CPU, for the first time, it proposes the multi-period discriminators(MPD), which have been proven to be a key design for high synthesis quality and have a great impact on the succeeding work su”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “The acoustic vocoder is separately trained with a design similar to Kumar et al. [2024], Lee et al. [2022], Cong et al. [2021] and Liu and Qian [2021].”
• eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer (2023, methodology): “Previously, Neural Text-to-Speech (NTTS) methods depended on training a separate component to generate mel-spectrograms from phonemized text [1–4] and another component (vocoder) to generate speech waveforms from mel-spectrograms [5–9].”
• Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, methodology): “Consequently, to reconstruct the input the speaker embedding is concatenated to the non-speaker features and the result is fed to the decoder, a BigVGAN (gil Lee et al., 2023) model, to generate the final speech.”