A Spectral Energy Distance for Parallel Speech Synthesis

By Alexey A. Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, Nal Kalchbrenner. 2020

84 citations recorded by the source corpus.

Speech synthesis is an important practical generative modeling problem that has seen great progress over the last few years, with likelihood-based autoregressive neural models now outperforming traditional concatenative systems. A downside of such autoregressive models is that they require executing tens of thousands of sequential operations per second of generated audio, making them ill-suited for deployment on specialized deep learning hardware. Here, we propose a new learning method that allows us to train highly parallel models of speech, without requiring access to an analytical likelihood function. Our approach is based on a generalized energy distance between the distributions of the generated and real audio. This spectral energy distance is a proper scoring rule with respect to the distribution over magnitude-spectrograms of the generated waveform audio and offers statistical consistency guarantees. The distance can be calculated from minibatches without bias, and does not involve adversarial learning, yielding a stable and consistent method for training implicit generative models. Empirically, we achieve state-of-the-art generation quality among implicit generative models, as judged by the recently-proposed cFDSD metric. When combining our method with adversarial techniques, we also improve upon the recently-proposed GAN-TTS model in terms of Mean Opinion Score as judged by trained human evaluators.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yossi Adi
5. Zhou Zhao
6. Yu Zhang
7. Yi Ren
8. Wei-Ning Hsu

In Representation & Self-Supervised Learning

1. Yossi Adi
2. Neil Zeghidour
3. Lior Wolf
4. Gabriel Synnaeve
5. William Chan
6. Ron J. Weiss
7. Devi Parikh
8. Yu Zhang

In Signal Processing

1. Xu Tan
2. Yossi Adi
3. Tao Qin
4. Tie-Yan Liu
5. Rongjie Huang
6. Yi Ren
7. Zhou Zhao
8. Frank K. Soong

Citations in the corpus

• WaveGrad: Estimating Gradients for Waveform Generation (2020, background): “, 2020a;b; McCarthy & Ahmed, 2020) and energy score (Gritsenko et al., 2020), variational auto-encoder models (Peng et al.”
• SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “The multi-scale spectral reconstruction loss follows the specifications described in [48]: where S st ( x ) denotes the t -th frame of a 64-bin mel-spectrogram computed with window length equal to s and hop length equal to s/ 4 .”
• High Fidelity Neural Audio Compression (2022, methodology): “For the frequency domain, we use a linear combination between the L1 and L2 losses over the mel-spectrogram using several time scales (Yamamoto et al., 2020b; Gritsenko et al., 2020).”
• High-Fidelity Audio Compression with Improved RVQGAN (2023, background): “Frequency domain reconstruction loss: while the mel-reconstruction loss [18] is known to improve stability, fidelity and convergence speed, the multi-scale spectral losses[41, 11, 15] encourage modeling of frequencies in multiple time-scales.”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “In the former category, Gritsenko et al. (2020) presents a variant of a model trained to produce STFT coefficients.”
• A Survey on Neural Speech Synthesis (2021, methodology): “WaveGAN [396] Voc melS FF−→wav ICASSP20 2019.10 DCA-Tacotron [17] AM ph AR−→melS AR−→wav ICASSP20 2019.10 WaveFlow [265] Voc melS AR−→wav ICML20 2019.12 SqueezeWave [426] Voc melS FF−→wav arXiv20 2020.01 AlignTTS [422] AM ch/ph FF−→melS FF−→wav ICASSP20 2020.03 RobuTrans [190] AM ph AR−→melS AR−→wav”
• AudioGen: Textually Guided Audio Generation (2022, methodology): “For the frequency domain loss, we use a linear combination between the L1 and L2 losses over the mel-spectrogram using several time scales (Yamamoto et al., 2020; Gritsenko et al., 2020).”
• Disentangling speech from surroundings in a neural audio codec (2022, methodology): “To train the model, we minimize a multi-scale spectral reconstruction loss for each step [25, 26]:”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, methodology): “Besides, We adopt Fréchet Deep Speech Distances (FDSD) (Gritsenko et al. 2020) to judge the quality of synthetic audio samples based on their distance to a reference set.”
• Cross-Scale Vector Quantization for Scalable Neural Speech Coding (2022, methodology): “Following [25], we calculate the mel spectra over a sequence of window-lengths between 64 and 2048.”
• Wave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis (2020, methodology): “Other approaches have eschewed probabilistic models and used GANs [10, 11], or carefully constructed spectral losses [12, 13, 14].”
• SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping (2022, methodology): “The weight for GED was 0.01 which is the same as that in the PriorGrad paper.”
• PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior (2021, methodology): “We chose the generalized energy distance (GED) [6] as L cycle because it mitigates the drawbacks of the point-to-point regression used in other spectral losses.”
• Upsampling Artifacts in Neural Audio Synthesis (2020, methodology): “It has been used, e.g., for music source separation [2], speech enhancement [22] and neural vocoding [14].”
• ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, methodology): “The use of iSTFT for neural speech synthesis was previously introduced [26, 27, 28] (including our own early attempts [26]).”
• High Fidelity Speech Regeneration with Application to Speech Enhancement (2021, methodology): “Inspired by the recently suggested spectral energy distance formulation of the spectral loss [16], the spectral loss is applied as part of a compound loss term:”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[63] proposed a method for training parallel vocoder based on the spectral generalized energy distance (GED) [58, 180, 188] between the generated and the real audio distribution.”
• Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “We used the same setups as [46] for other settings except the removal of the generalized energy distance (GED) loss [69] as we observed no impact on the quality.”