Wave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis

By Ron J. Weiss, R. Skerry-Ryan, Eric Battenberg, Soroosh Mariooryad, Diederik P. Kingma. 2020

111 citations recorded by the source corpus.

We describe a sequence-to-sequence neural network which directly generates speech waveforms from text inputs. The architecture extends the Tacotron model by incorporating a normalizing flow into the autoregressive decoder loop. Output waveforms are modeled as a sequence of non-overlapping fixed-length blocks, each one containing hundreds of samples. The interdependencies of waveform samples within each block are modeled using the normalizing flow, enabling parallel training and synthesis. Longer-term dependencies are handled autoregressively by conditioning each flow on preceding blocks. This model can be optimized directly with maximum likelihood, with-out using intermediate, hand-designed features nor additional loss terms. Contemporary state-of-the-art text-to-speech (TTS) systems use a cascade of separately learned models: one (such as Tacotron) which generates intermediate features (such as spectrograms) from text, followed by a vocoder (such as WaveRNN) which generates waveform samples from the intermediate features. The proposed system, in contrast, does not use a fixed intermediate representation, and learns all parameters end-to-end. Experiments show that the proposed model generates speech with quality approaching a state-of-the-art neural TTS system, with significantly improved generation speed.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Wei-Ning Hsu
6. Sheng Zhao
7. Lei He
8. Yu Zhang

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Model One-Stage Training AR/NAR Modeling Architecture Char2Wav [309] N AR Seq2Seq RNN ClariNet [263] N AR Flow CNN FastSpeech 2s [286] Y NAR GAN Self-Att/CNN EATS [67] Y NAR GAN CNN Wave-Tacotron [379] Y AR Flow CNN/RNN/Hybrid EfficientTTS-Wav [229] Y NAR GAN CNN VITS [157] Y NAR VAE+Flow CNN/Self-A”
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, methodology): “In speech synthesis, Hsu et al. (2019) and Zhang et al. (2019) combine Tacotron 2 and VAEs to learn speech style and prosody.”
• WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis (2021, methodology): “For Wave-Tacotron [33] and the proposed WaveGrad 2 models, we used a subset of training set which included all the audio of the test speaker.”
• fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, methodology): “We additionally, report Mel Cepstral Distortion (MCD), Mel Spectral Distortion (MSD), and CER to evaluate both the overall similarity to the target speech and content intelligibility (Weiss et al., 2021).”
• Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, methodology): “WaveFlow uses autoregression to handle short-range dependencies and a non-autoregressive architecture to model long-range correlations, while Wave-Tacotron uses autoregres-sion for long-range dependencies and a non-autoregressive model for short-range dependencies.”
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention (2021, methodology): “Both EATS (Weiss et al., 2020) and Fastspeech 2s (Ren et al., 2020) are non-AR models, where various adversarial feedbacks and auxiliary prediction losses are used respectively.”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Most end-to-end methods still rely on generating mel-spectrogram for text-speech alignment, and a spectrogram-free flow-based method is investigated in Wave-Tacotron [123] by simply maximizing likelihood.”
• WavThruVec: Latent speech representation as intermediate features for neural speech synthesis (2022, background): “This approach makes it possible to learn hidden, highdimensional representations throughout the network and it has led to several promising results [16, 17, 18, 19].”
• JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech (2022, methodology): “On the other hand, end-to-end text-to-speech (E2E-TTS) [5], [13], [14], [15], [16] is a recent research trend in which a speech waveform is directly generated from an input text in a single stage without distinction between an acoustic feature generator and a neural vocoder.”
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis (2021, background): “Integrating acoustic model and vocoder into an end-to-end model is another possible solution [4, 15, 16, 17], but this kind of models are notoriously difficult to train and extra Mel-spectrum based loss has to be adopted to stabilize the convergence process.”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “Neural speech synthesis [9, 10, 11, 12, 13, 14, 15, 16, 17, 18] have showed close to human quality in terms of naturalness and audio fidelity in the past years, with cascaded or end-to-end transformer or LSTM architectures.”
• Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, unknown): “Generally, with a well-trained neural acoustic model [2, 5, 6, 7] and a neural vocoder [8, 9, 10, 11], or alternatively using fully end-to-end models [12, 13, 14] which directly construct wave signals from text input, it is able to synthesize high-quality neutral speech.”
• NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, background): “End-to-end training methods has been recently proposed to tackle such issues (Bi ´ nkowski et al., 2020; Weiss et al., 2021; Donahue et al., 2021; Kim et al., 2021).”