DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

By Songxiang Liu, Dan Su, Dong Yu. 2022

82 citations recorded by the source corpus.

Denoising diffusion probabilistic models (DDPMs) are expressive generative models that have been used to solve a variety of speech synthesis problems. However, because of their high sampling costs, DDPMs are difficult to use in real-time speech processing applications. In this paper, we introduce DiffGAN-TTS, a novel DDPM-based text-to-speech (TTS) model achieving high-fidelity and efficient speech synthesis. DiffGAN-TTS is based on denoising diffusion generative adversarial networks (GANs), which adopt an adversarially-trained expressive model to approximate the denoising distribution. We show with multi-speaker TTS experiments that DiffGAN-TTS can generate high-fidelity speech samples within only 4 denoising steps. We present an active shallow diffusion mechanism to further speed up inference. A two-stage training scheme is proposed, with a basic TTS acoustic model trained at stage one providing valuable prior information for a DDPM trained at stage two. Our experiments show that DiffGAN-TTS can achieve high synthesis performance with only 1 denoising step.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Nima Mesgarani
3. Zhou Zhao
4. Cong Han
5. Rongjie Huang
6. Yi Ren
7. Jinglin Liu
8. Li Deng

In Signal Processing

1. Nima Mesgarani
2. Rongjie Huang
3. Dong Yu
4. Zhou Zhao
5. Yi Ren
6. Cong Han
7. Jinglin Liu
8. Dongchao Yang

Citations in the corpus

• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
• InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt (2023, methodology): “Following previous work [72], we also adopt them as one of the metrics for speech quality.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “Some recent works [26, 40] choose to parameterize the denoising model through directly predicting x0 with a neural network fθ .”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Inspired by [126] that adopts GAN to model the denoising distribution which enables larger step size and less steps, DiffGAN-TTS [60] applies a pretrained GAN as the acoustic generator for acceleration.”
• LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[91] Songxiang Liu, Dan Su, and Dong Yu.”
• Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “This strategy was applied to TTS, especially to predict a log-mel spectrogram given an input text [51].”