Fre-GAN: Adversarial Frequency-consistent Audio Synthesis

By Ji-Hoon Kim, Sang-Hoon Lee, Ji-Hyun Lee, Seong-Whan Lee. 2021

61 citations recorded by the source corpus.

Although recent works on neural vocoder have improved the quality of synthesized audio, there still exists a gap between generated and ground-truth audio in frequency space. This difference leads to spectral artifacts such as hissing noise or reverberation, and thus degrades the sample quality. In this paper, we propose Fre-GAN which achieves frequency-consistent audio synthesis with highly improved generation quality. Specifically, we first present resolution-connected generator and resolution-wise discriminators, which help learn various scales of spectral distributions over multiple frequency bands. Additionally, to reproduce high-frequency components accurately, we leverage discrete wavelet transform in the discriminators. From our experiments, Fre-GAN achieves high-fidelity waveform generation with a gap of only 0.03 MOS compared to ground-truth audio while outperforming standard models in quality.

Subfields: Biomedical Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Xu Tan

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Lei He
5. Sheng Zhao
6. Frank K. Soong
7. Yanqing Liu
8. Xuejiao Tan

Citations in the corpus

• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Discrete wavelet transform is used to replace average sampling in the discriminators as [18] to reproduce highfrequency components accurately.”
• A Survey on Neural Speech Synthesis (2021, methodology): “WaveGAN [402] Mel-Spectrogram NAR GAN CNN HiFi-GAN [174] Mel-Spectrogram NAR GAN Hybrid/CNN VocGAN [408] Mel-Spectrogram NAR GAN CNN GED [96] Linguistic Feature NAR GAN CNN Fre-GAN [161] Mel-Spectrogram NAR GAN CNN Wave-VAE [268] Mel-Spectrogram NAR VAE CNN WaveGrad [41] Mel-Spectrogram NAR Diffusio”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “The predicted frame-level feature is transformed to raw waveform by the neural vocoders with adversarial training [20, 21, 22, 23].”
• Avocodo: Generative Adversarial Network for Artifact-free Vocoder (2022, methodology): “…methods, such as the average pooling used in (Kumar et al. 2019; Kong, Kim, and Bae 2020; Yang et al. 2020) or the equally spaced sampling used in (Kong, Kim, and Bae 2020; Kim et al. 2021; Jang et al. 2021), are easy to implement and efficient for obtaining band-limited speech waveforms.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “In addition, we can synthesize speech di ff erently with di ff erent Gaussian noises and control the duration by multiplying the duration by a specific value.”