UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

By Won Jang, D. Lim, Jaesam Yoon, Bongwan Kim, Juntae Kim. 2021

200 citations recorded by the source corpus.

Most neural vocoders employ band-limited mel-spectrograms to generate waveforms. If full-band spectral features are used as the input, the vocoder can be provided with as much acoustic information as possible. However, in some models employing full-band mel-spectrograms, an over-smoothing problem occurs as part of which non-sharp spectrograms are generated. To address this problem, we propose UnivNet, a neural vocoder that synthesizes high-fidelity waveforms in real time. Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lei He
2. Sheng Zhao
3. Cong Han
4. Nima Mesgarani
5. Xu Tan
6. Yanqing Liu
7. Tao Qin
8. Boris Ginsburg

In Speech & Audio

1. Lei He
2. Sheng Zhao
3. Yanqing Liu
4. Wei Ping
5. Boris Ginsburg
6. Hubert Siuzdak
7. D. Lim
8. Congyi Wang

In Signal Processing

1. Lei He
2. Sheng Zhao
3. Yanqing Liu
4. Cong Han
5. Hubert Siuzdak
6. Wei Ping
7. Nima Mesgarani
8. Boris Ginsburg

Citations in the corpus

• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “Our baseline uses a diffusion-based spectrogram decoder and a separately trained UnivNet [51] vocoder, as proposed in [21].”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “[47] along with the LSGAN loss functions [48] for decoder training, and incorporate the truncated pointwise relativistic loss function [49] to enhance sound quality (see Appendix F and G for details).”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “However, spectrograms of generated audio can still appear blurry, exhibiting over-smoothing artifacts in high frequencies[16].”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “Discriminator We employ the multi-period discriminator (MPD) as defined by Kong et al. (2020), and multi-resolution discriminator (MRD) (Jang et al., 2021).”
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “We also train UnivNet-c32 (Jang et al., 2021) on LibriTTS train-full using the open-source implementation (note that UnivNet uses the 24,000Hz sampling rate and the 100-band log-mel spectrogram with [0, 12] kHz frequency by default).”
• JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech (2022, methodology): “There are lots of research on neural vocoder as well and some of the famous, widely used include [6], [7], normalizing flow based one [8] and generative adversarial network (GAN) based ones [9], [10], [11], [12].”
• Improve GAN-based Neural Vocoder using Truncated Pointwise Relativistic Least Square GAN (2022, methodology): “Above all, we combine the original MSE loss with the pointwise relativistic discrepancy loss to formulate our T-PRLSGAN as follows: where L is the vanilla LSGAN loss and 𝜆 𝑟𝑒𝑙 is experimentally chosen as 1.0 in FB-MelGAN[34] and UnivNet[5] and 0.1 in other vocoders.”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “The predicted frame-level feature is transformed to raw waveform by the neural vocoders with adversarial training [20, 21, 22, 23].”
• Avocodo: Generative Adversarial Network for Artifact-free Vocoder (2022, methodology): “…methods, such as the average pooling used in (Kumar et al. 2019; Kong, Kim, and Bae 2020; Yang et al. 2020) or the equally spaced sampling used in (Kong, Kim, and Bae 2020; Kim et al. 2021; Jang et al. 2021), are easy to implement and efficient for obtaining band-limited speech waveforms.”