MOSNet: Deep Learning based Objective Assessment for Voice Conversion

By Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Eric Wang, Junichi Yamagishi, Yu Tsao, Hsin-Min Wang. 2019

373 citations recorded by the source corpus.

Existing objective evaluation metrics for voice conversion (VC) are not always correlated with human perception. Therefore, training VC models with such criteria may not effectively improve naturalness and similarity of converted speech. In this paper, we propose deep learning-based assessment models to predict human ratings of converted speech. We adopt the convolutional and recurrent neural network models to build a mean opinion score (MOS) predictor, termed as MOSNet. The proposed models are tested on large-scale listening test results of the Voice Conversion Challenge (VCC) 2018. Experimental results show that the predicted scores of the proposed MOSNet are highly correlated with human MOS ratings at the system level while being fairly correlated with human MOS ratings at the utterance level. Meanwhile, we have modified MOSNet to predict the similarity scores, and the preliminary results show that the predicted scores are also fairly correlated with human ratings. These results confirm that the proposed models could be used as a computational evaluator to measure the MOS of VC systems to reduce the need for expensive human rating.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Sheng Zhao
7. Lei He
8. Nima Mesgarani

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Junichi Yamagishi
5. Frank K. Soong
6. Lei He
7. Nima Mesgarani
8. Sheng Zhao

Citations in the corpus

• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “The training target of MOSNet [27] is to predict the average MOS of each utterance, while MBNet [28] also considers the judging bias of each rater.”
• NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, background): “1) Judgment Guideline: While there are some objective metrics to measure the quality gap between the generated speech and human recordings, such as PESQ [30], STOI [31], SI-SDR [32], they are not reliable to measure the perception quality in TTS [33], [34].”
• Dnsmos: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2020, background): “Recently, Deep Neural Networks (DNN) based approaches have been proposed to estimate the speech quality scores [6, 7, 8, 9].”
• ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild (2022, methodology): “Some form of objective scores, e.g. derived using MOSNet [58], might also be used as a proxy for subjective scores.”
• HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, methodology): “Following Lo et al. (2019), we use listening evaluation results released by Voice Conversion Challenge 2018 (Lorenzo-Trueba et al., 2018) for model training.”
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem (2021, methodology): “Besides the STOI, SAR, SDR and SIR, we use an additional open-source MOSNet [30] results to provide some rough simulation of subjective evaluation about the speech quality.”
• VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, methodology): “• Regarding TTS, we measure intelligibility with CER and quality using neural-predicted MOS quality score MOSNet [31,32].”
• MBNET: MOS Prediction for Synthesized Speech with Mean-Bias Network (2021, methodology): “• To the best of our knowledge, all the previous works [13, 14, 16, 17] leveraged the utterance-level MOS as the training target, and discarded the detailed judge scores.”
• Deep Learning Based Assessment of Synthetic Speech Naturalness (2020, methodology): “MosNet [18], was also tested on the VCC 2016 dataset with a correlation of 0.”
• StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion (2021, methodology): “We use predicted mean opinion score (pMOS) from MOSNet [26] to conduct the ablation study.”
• Expressive TTS Training With Frame and Style Reconstruction Loss (2020, background): “[74] propose deep learning-based assessment models to predict human ratings of converted speech.”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, methodology): “[285] propose MOSNet, another non-intrusive assessment technique based on deep neural networks, that learns to predict human MOS ratings.”
• Utilizing Self-supervised Representations for MOS Prediction (2021, methodology): “Also, following the previous work [9], the segmentlevel MSE in (11) becomes frame-level MSE, which enforces all frames produce the same score.”
• Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio (2023, background): “Several works have explored training neural networks for speech MOS estimation [4, 5, 6, 7, 8, 9, 10, 11].”