Revisiting Over-Smoothness in Text to Speech

By Yi Ren, Xu Tan, Tao Qin, Zhou Zhao, Tie-Yan Liu. 2022

77 citations recorded by the source corpus.

Non-autoregressive text to speech (NAR-TTS) models have attracted much attention from both academia and industry due to their fast generation speed. One limitation of NAR-TTS models is that they ignore the correlation in time and frequency domains while generating speech mel-spectrograms, and thus cause blurry and over-smoothed results. In this work, we revisit this over-smoothing problem from a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distributions and the capability of modeling methods. Both simplifying data distributions and improving modeling methods can alleviate the problem. Accordingly, we first study methods reducing the complexity of data distributions. Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods. Based on these studies, we find that 1) methods that provide additional condition inputs reduce the complexity of data distributions to model, thus alleviating the over-smoothing problem and achieving better voice quality. 2) Among advanced modeling methods, Laplacian mixture loss performs well at modeling multimodal distributions and enjoys its simplicity, while GAN and Glow achieve the best voice quality while suffering from increased training or model complexity. 3) The two categories of methods can be combined to further alleviate the over-smoothness and improve the voice quality. 4) Our experiments on the multi-speaker dataset lead to similar conclusions as above and providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rongjie Huang
2. Jinglin Liu
3. Zhou Zhao
4. Yi Ren
5. Chenye Cui
6. Sang-Hoon Lee
7. Huadai Liu
8. Seong-Whan Lee

In Signal Processing

1. Rongjie Huang
2. Jinglin Liu
3. Zhou Zhao
4. Yi Ren
5. Chenye Cui
6. Sang-Hoon Lee
7. Huadai Liu
8. Minki Kang

Citations in the corpus

• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, background): “Qualitative visualization demonstrates that diffusion models help to overcome the over-smoothness issues in previous works [18] by modeling the high-frequency components in detail.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “Inspired by [37], we adopt structural similarity index (SSIM) loss in training TTS models:”
• DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, methodology): “This design is inspired by multi-length window discriminators [44].”