Probability density distillation with generative adversarial networks for high-quality parallel waveform generation

By Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim. 2019

58 citations recorded by the source corpus.

This paper proposes an effective probability density distillation (PDD) algorithm for WaveNet-based parallel waveform generation (PWG) systems. Recently proposed teacher-student frameworks in the PWG system have successfully achieved a real-time generation of speech signals. However, the difficulties optimizing the PDD criteria without auxiliary losses result in quality degradation of synthesized speech. To generate more natural speech signals within the teacher-student framework, we propose a novel optimization criterion based on generative adversarial networks (GANs). In the proposed method, the inverse autoregressive flow-based student model is incorporated as a generator in the GAN framework, and jointly optimized by the PDD mechanism with the proposed adversarial learning method. As this process encourages the student to model the distribution of realistic speech waveform, the perceptual quality of the synthesized speech becomes much more natural. Our experimental results verify that the PWG systems with the proposed method outperform both those using conventional approaches, and also autoregressive generation systems with a well-trained teacher WaveNet.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yoshua Bengio
6. Sheng Zhao
7. Lior Wolf
8. Zhou Zhao

In Generative Models

1. Xu Tan
2. Wei Ping
3. Tao Qin
4. Tie-Yan Liu
5. Yoshua Bengio
6. Karen Simonyan
7. Aaron C. Courville
8. Frank K. Soong

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Lior Wolf
5. Wei Ping
6. Frank K. Soong
7. Sheng Zhao
8. Yossi Adi

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Wei Ping
6. Sheng Zhao
7. Yossi Adi
8. Ryuichi Yamamoto

Citations in the corpus

• GAN Memory with No Forgetting (2020, background): “Generative adversarial networks (GANs) GANs have shown increasing power to synthesize highly realistic observations [32, 45, 51, 9, 33, 34], and have found wide applicability in various fields [39, 1, 19, 81, 83, 84, 12, 89, 37].”
• A Survey on Neural Speech Synthesis (2021, background): “Except for the regular GAN losses such as WGAN-GP [94], hinge-loss GAN [194], and LS-GAN [225], other specific losses such as STFT loss [10, 395] and feature matching loss [178] are also leveraged.”
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “We use the pretrained WaveNet vocoder model provided by Yamamoto (2019) and train the Text2mel model with the required data preprocessing performed.”
• Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, background): “Similarly to [24, 25], we define the STFT loss to be the sum of the spectral convergence (sc) loss and the magnitude loss as follows,”
• Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram (2019, methodology): “We also investigated ClariNet with adversarial loss as a hybrid approach of GAN and density distillation [11].”
• End-to-End Adversarial Text-to-Speech (2020, methodology): “lso implies that the spectrogram generation model and the vocoder have to be trained sequentially. 6 Spectrogram prediction losses have been used extensively for feed-forward audio prediction models (Yamamoto et al., 2019; 2020; Yang et al., 2020; Arık et al., 2018; Engel et al., 2020; Wang et al., ”
• WaveFlow: A Compact Flow-based Model for Raw Audio (2019, methodology): “Most recently, flow-based models have been successfully applied for parallel waveform synthesis with comparable fidelity as autoregressive models (van den Oord et al., 2018; Ping et al., 2019; Yamamoto et al., 2019b; Prenger et al., 2019; Kim et al., 2019; Serrà et al., 2019).”
• Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, methodology): “To solve this problem, we adopt the multi-resolution STFT loss, which has been proven to be more effective to measure the difference between fake and real speech [21, 23, 24].”
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis (2021, methodology): “As for the proposed VAE for speech reconstruction, we adopt multi-resolution STFT loss [10, 20, 21] to measure the distance between natural speech w and generated speech ŵ in frequency domain.”
• Improving LPCNET-Based Text-to-Speech with Linear Prediction-Structured Mixture Density Network (2020, background): “As the STFT loss function is able to effectively capture the time-frequency distribution of the realistic speech waveform, especially for the harmonic components [12, 13], the entire training process becomes more efficient.”
• FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction (2020, methodology): “Additionally, our implementation of Parallel WaveNet [10] requires 8 cores to achieve the similar synthesis speed.”
• Improved Parallel Wavegan Vocoder with Perceptually Weighted Spectrogram Loss (2021, methodology): “[16] first proposed spectral convergence and log-scale STFT magnitude losses, and our previous work proposed combing these in a multi-resolution form [9].”
• On Leveraging Pretrained GANs for Generation with Limited Data (2020, unknown): “…image (Hoffman et al., 2017; Ledig et al., 2017; Ak et al., 2019), text (Lin et al., 2017; Fedus et al., 2018; Wang & Wan, 2018; Wang et al., 2019), video (Mathieu et al., 2015; Wang et al., 2017; 2018a; Chan et al., 2019), and audio (Engel et al., 2019; Yamamoto et al., 2019; Kumar et al., 2019).”