HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

By Jungil Kong, Jaehyeon Kim, Jaekyoung Bae. 2020

2,760 citations recorded by the source corpus.

Several recent work on speech synthesis have employed generative adversarial networks (GANs) to produce raw waveforms. Although such methods improve the sampling efficiency and memory usage, their sample quality has not yet reached that of autoregressive and flow-based generative models. In this work, we propose HiFi-GAN, which achieves both efficient and high-fidelity speech synthesis. As speech audio consists of sinusoidal signals with various periods, we demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality. A subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method demonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen speakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

1. Xu Tan
2. Wei Ping
3. Yi Ren
4. Zhou Zhao
5. Tao Qin
6. Tie-Yan Liu
7. Yossi Adi
8. Jinglin Liu

Citations in the corpus

• On Fast Sampling of Diffusion Probabilistic Models (2021, methodology): “Most recently, these models have obtained the state-of-the-art results in several important domains, including image synthesis (Ho et al., 2020; Song et al., 2020b; Dhariwal and Nichol, 2021), audio synthesis (Kong et al., 2020b; Chen et al., 2020), and 3-”
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, background): “It should be noted that AudioLM is able to synthesize speech based on audio codecs without training an additional vocoder such as HifiGAN [Kong et al., 2020].”
• SoundStream: An End-to-End Neural Audio Codec (2021, neutral): “HiFiGAN [16] takes a similar approach but it applies discriminators to both multiple scales and multiple periods of the audio samples.”
• High Fidelity Neural Audio Compression (2022, methodology): “We discuss the impact on the processing time in Section 4.6.”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, background): “…waveform with these disentangled representations, leveraging information bottleneck [16, 17], various supervised losses, and adversarial training [18] to enhance disentanglement; 2) we propose a factorized diffusion model, which generates the factorized speech representations of duration,…”
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “The output of a HiFi-GAN based decoder block [53] is fed into BigVGAN vocoder [54] to predict the waveform.”
• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “Audio is represented as 80-dimensional Mel spectrograms and are converted to waveform using a HiFi-GAN vocoder (Kong et al., 2020).”
• FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “Building upon the progress in text-to-speech synthesis [5], adversarial training losses are employed to enhance reconstruction quality.”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Two common methods for vocoding semantic tokens, namely HifiGAN and AudioLM, are used as baselines in our evaluation.”
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “We replace the mel-discriminator in [6] with the multi-period discriminator (MPD) [30] and multi-resolution discriminator (MRD)”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “We use the multi-period discriminator [18], and a complex multi-scale STFT discriminator.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “The predicted mel-spectrograms are transformed into audio samples using pre-trained HiFi-GAN V1 8 [31].”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “On the other hand, iSTFTNet (Kaneko et al., 2022) proposes modifications to HiFi-GAN, enabling it to return magnitude and phase spectrum.”
• Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, background): “One of the state-of-the-art neural vocoders, HiFi-GAN [29], is integrated into our codec, and we argue that the combination of a separate powerful vocoder and a well-trained encoder will attain the highest quality.”
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec (2023, methodology): “In this study, we use three discriminators: A multi-scale STFTbased (MS-STFT) discriminator, which is used on Encodec [7]; a multi-period discriminator (MPD) and a multi-scale discriminator (MSD) from HiFi-GAN vocoder [22].”
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Speech/Audio Synthesis With the rapid development and application of neural networks, speech and audio synthesis have made tremendous progress with different network frameworks, such as WaveNet [Oord et al., 2016], HiFi-GAN [Kong et al., 2020a], and Diffwave Kong et al. [2020b].”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “Recently, the deep neural network-based Text-To-Speech (TTS) synthesis models have shown remarkable performance on both quality and speed, thanks to the progress on the generative modeling [1, 2], non-autoregressive acoustic models [3], and the powerful neural vocoder [4].”
• AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “While WaveNet inspired more computationally efficient alternatives such as WaveRNN [2] or parallel WaveNet [20], a significant paradigm shift occurred with the introduction of adversarial audio generation [3], [4], [21], which enables high fidelity generation without any autoregressive component.”
• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “For adversarial training, we combine the same multi-scale and multi-period discriminators as [17, 15].”
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, background): “As for the vocoders, generative adversarial network (GAN) [6] based vocoders, such as multi-band MelGAN [7] and HifiGAN [8], are widely used for their high quality of speech and fast generation speed.”