ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

By Wei Ping, Kainan Peng, Jitong Chen. 2018

365 citations recorded by the source corpus.

In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, methodology): “ClariNet (Ping et al., 2019) achieves similar audio quality using a simple Gaussian distribution instead and thus having a closed-form loss function, eliminating the need for Monte-Carlo sampling.”
Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “All three require compound loss functions to improve audio quality or problems with mode collapse [9, 7, 8].”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “…the waveform conditioned on the aligned linguistic features (van den Oord et al., 2016; Arık et al., 2017a), the mel spectrogram from a text-to-spectrogram model (Ping et al., 2018; Shen et al., 2018), or the hidden states within text-to-wave architecture (Ping et al., 2019; Donahue et al., 2020).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “There has been a plethora of research into non-autoregressive models for audio generation, including normalizing flows such as inverse autoregressive flows (Oord et al., 2018; Ping et al., 2019), generative flows (Prenger et al.”
Neural Speech Synthesis with Transformer Network (2018, background): “Recently, ClariNet (Ping, Peng, and Chen 2018), a fully convolutional text-to-wave neural architecture, is proposed to enable the fast end-to-end training from scratch.”
DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Although ideally having advantages over two-stage systems, different end-to-end models face their own challenges and limitations: 1) no significant improvement over two-stage models in terms of voice quality, such as FastSpeech 2s [3], EATS [6], and WAVE-TACOTRON [8]; 2) slow inference speed due to ”
A Survey on Neural Speech Synthesis (2021, methodology): “Similarly, ClariNet [263] uses IAF and teacher distillation, and leverages a closed-form KL divergence to simplify and stabilize the distillation process.”
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (2020, methodology): “(2020) proposed multi-resolution STFT loss function to improve and stabilize GAN training and achieved better parameter efficiency and less training time than an IAF model, ClariNet (Ping et al., 2018).”
Semi-supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis (2018, background): “We are now able to produce natural prosody with high audio fidelity using a much simplified voice building pipeline [1, 2, 3].”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, neutral): “For example, the probability distillation methods ClariNet [26] and Parallel WaveNet [34] minimize the distance between spectrogram magnitudes of real and synthesized speech in addition to their main distillation loss.”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “Parallel Wavenet (Oord et al., 2017) and Clarinet (Ping et al., 2018) distill a trained auto-regressive decoder into a ﬂow-based convolutional student model.”
It's Raw! Audio Generation with State-Space Models (2022, background): “Additionally, it serves as the backbone for several families of non-AR generative models of audio in both the conditional and unconditional settings: (i) Distillation: Parallel WaveNet [41] and ClariNet [32] distill parallelizable flow models from a teacher WaveNet model.”
GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, background): “Their subsequent knowledge-distilled variants are mostly geared to lessen the computational overhead for training and inference [3, 4].”
Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram (2019, methodology): “As baseline systems, we used both the autoregressive Gaussian WaveNet and the parallel one (i.e., ClariNet) [10,11].”
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “Compared with other text-towaveform models such as ClariNet [18] that jointly train autoregressive acoustic model and nonautoregressive vocoder, FastSpeech 2s employs the fully non-autoregressive architecture.”
High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “Alternatively, an invertible feed-forward model can be trained by distilling an autoregressive model using probability density distillation (van den Oord et al., 2018; Ping et al., 2019), which enables it to focus on particular modes.”
FloWaveNet : A Generative Flow for Raw Audio (2018, methodology): “The recently suggested ClariNet (Ping et al., 2018) presented an alternative formulation by using a single Gaussian distribution with a closed-form Kullback-Leibler (KL) divergence, contrasting with the high-variance Monte Carlo approximation from Parallel WaveNet.”
End-to-End Adversarial Text-to-Speech (2020, positive): “In particular, it does not rely on autoregressive sampling or teacher forcing, avoiding issues like exposure bias [6, 49] and reduced parallelism at inference time, or the complexities introduced by distillation to a more efﬁcient feed-forward model after the fact [45, 61].”