DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021

By Yanqing Liu, Zhihang Xu, G. Wang, Kuan-Hen Chen, Bohan Li, Xu Tan, Jinzhu Li, Lei He, Sheng Zhao. 2021

63 citations recorded by the source corpus.

This paper describes the Microsoft end-to-end neural text to speech (TTS) system: DelightfulTTS for Blizzard Challenge 2021. The goal of this challenge is to synthesize natural and high-quality speech from text, and we approach this goal in two perspectives: The first is to directly model and generate waveform in 48 kHz sampling rate, which brings higher perception quality than previous systems with 16 kHz or 24 kHz sampling rate; The second is to model the variation information in speech through a systematic design, which improves the prosody and naturalness. Specifically, for 48 kHz modeling, we predict 16 kHz mel-spectrogram in acoustic model, and propose a vocoder called HiFiNet to directly generate 48 kHz waveform from predicted 16 kHz mel-spectrogram, which can better trade off training efficiency, modelling stability and voice quality. We model variation information systematically from both explicit (speaker ID, language ID, pitch and duration) and implicit (utterance-level and phoneme-level prosody) perspectives: 1) For speaker and language ID, we use lookup embedding in training and inference; 2) For pitch and duration, we extract the values from paired text-speech data in training and use two predictors to predict the values in inference; 3) For utterance-level and phoneme-level prosody, we use two reference encoders to extract the values in training, and use two separate predictors to predict the values in inference. Additionally, we introduce an improved Conformer block to better model the local and global dependency in acoustic model. For task SH1, DelightfulTTS achieves 4.17 mean score in MOS test and 4.35 in SMOS test, which indicates the effectiveness of our proposed system

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Lei He
6. Yanqing Liu
7. Yichong Leng
8. Bohan Li

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Tao Qin
4. Tie-Yan Liu
5. Lei He
6. Yanqing Liu
7. Bohan Li
8. Yichong Leng

Citations in the corpus

• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, background): “Text-to-speech systems [2, 3, 4, 5, 6, 8, 9, 10, 11] aim to generate natural voice with both high quality and diversity.”
• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “• The acoustic model [4] has four variance information modules: duration predictor, pitch predictor, utterance-level acoustic predictor and phone-level acoustic predictor.”
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, background): “Neural text to speech (TTS) [1] models can synthesize high quality human voice when being trained with a large amount of single-speaker or multi-speaker datasets [2, 3, 4, 5, 6, 7, 8, 9].”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “Neural speech synthesis [9, 10, 11, 12, 13, 14, 15, 16, 17, 18] have showed close to human quality in terms of naturalness and audio fidelity in the past years, with cascaded or end-to-end transformer or LSTM architectures.”