End-to-End Adversarial Text-to-Speech

By Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, Karen Simonyan. 2020

197 citations recorded by the source corpus.

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

Subfields: Artificial Intelligence, Generative Models, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, background): “, 2018), or the hidden states within the text-to-wave architecture (Ping et al., 2019; Donahue et al., 2020).”
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “Several recent works have advanced towards human-level TTS [3, 4, 5] using techniques like BERT pre-training [4, 40, 7] and E2E training [32, 5] with differentiable duration modeling [41, 42].”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30”
DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “Although ideally having advantages over two-stage systems, different end-to-end models face their own challenges and limitations: 1) no significant improvement over two-stage models in terms of voice quality, such as FastSpeech 2s [3], EATS [6], and WAVE-TACOTRON [8]; 2) slow inference speed due to ”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “End-to-end TTS models have recently been proposed, such as Ren et al. (2021), Donahue et al. (2021), Weiss et al. (2021), Kim et al. (2021), and Chen et al. (2021b).”
A Survey on Neural Speech Synthesis (2021, methodology): “Replacing Attention with Duration FastSpeech [284], DurIAN [411], EATS [67] Glow-TTS [156], [190, 19, 355, 422, 73, 298]”
Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “Recently, several works, i.e., FastSpeech 2s (Ren et al., 2021) and EATS (Donahue et al., 2021), have proposed efﬁcient end-to-end training methods such as training over short audio clips rather than entire waveforms, leveraging a mel-spectrogram decoder to aid text representation learning, and desi”
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech (2021, methodology): “• As in (Bi ´ nkowski et al., 2020; Donahue et al., 2021 Once trained, Grad-TTS enables the trade-off between quality and inference speed due to the ability to vary the number of steps N the decoder takes to solve ODE (13) at inference.”
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “A concurrent work called EATS (Donahue et al., 2020) (4)We do not transform energy using CWT since energy is not as highly variable as pitch on LJSpeech dataset, and we do not observe gains when using it.”
NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “4) Compared to previous fully end-to-end TTS systems such as EATS [19], FastSpeech 2s [18], and VITS [15], we bridge the quality gap to recordings with advanced model designs to closely match the prior and posterior in the VAE framework.”
ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “Text-to-speech (TTS) models convert input text or phoneme sequence into mel-spectrogram (e.g., Tacotron [50], FastSpeech [36]), which is then transformed to waveform using a separately trained vocoder [9, 19], or directly generate waveform from text (e.g., EATS [6] and VITS [17]).”
Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “…Works 2.1 Speech/Singing Synthesis TTS or SVS models typically convert input text into mel-spectrogram (e.g., Tacotron [40], Fast-Speech [33]), which is then transformed to waveform using a separately trained vocoder [20, 12], or directly generate waveform from text (e.g., EATS [10] and VITS [19]).”
WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis (2021, methodology): “More recent work [7, 24] has focused on applying non-autoregressive models to end-to-end TTS.”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Donahue et al. [47] proposed an end-to-end TTS model EATS based on GAN-TTS [17], which directly synthesized speech non-autoregressively using GAN.”
FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “Neural speech synthesis [9, 10, 11, 12, 13, 14, 15, 16, 17, 18] have showed close to human quality in terms of naturalness and audio fidelity in the past years, with cascaded or end-to-end transformer or LSTM architectures.”
JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech (2022, methodology): “On the other hand, end-to-end text-to-speech (E2E-TTS) [5], [13], [14], [15], [16] is a recent research trend in which a speech waveform is directly generated from an input text in a single stage without distinction between an acoustic feature generator and a neural vocoder.”
Wave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis (2020, background): “, diffusion probabilistic models [34], or GANs [11, 19], which can be more easily optimized in a mode-seeking fashion that is likely to be more efficient than modeling the full data distribution.”