Deep Learning Based Assessment of Synthetic Speech Naturalness

By Gabriel Mittag, Sebastian Möller. 2020

85 citations recorded by the source corpus.

In this paper, we present a new objective prediction model for synthetic speech naturalness. It can be used to evaluate Text-To-Speech or Voice Conversion systems and works language independently. The model is trained end-to-end and based on a CNN-LSTM network that previously showed to give good results for speech quality estimation. We trained and tested the model on 16 different datasets, such as from the Blizzard Challenge and the Voice Conversion Challenge. Further, we show that the reliability of deep learning-based naturalness prediction can be improved by transfer learning from speech quality prediction models that are trained on objective POLQA scores. The proposed model is made publicly available and can, for example, be used to evaluate different TTS system configurations.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Lei He
4. Zhuo Chen
5. Yu Wu
6. Sheng Zhao
7. Sanyuan Chen
8. Xu Tan

In Signal Processing

1. Jinyu Li
2. Lei He
3. Shujie Liu
4. Zhuo Chen
5. Yu Wu
6. Sanyuan Chen
7. Sheng Zhao
8. Yanqing Liu

Citations in the corpus

• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “SN score of generated speech is measured by the open-source NISQA (Mittag and Möller, 2021).”
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Finally, to better verify our proposed VALL-E X systems, adopt the open-source NISQA 3 [Mittag and Möller, 2021] (the NISQA-TTS model) to evaluate the naturalness of the synthetic speech and further perform the human evaluation with manual scoring on the generated speech, e.g., mean opinion score…”
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We evaluate the Natural-ness with the open-source NISQA-TTS (Mittag and Möller, 2020 Table 3 shows the experimental results of TTS.”