UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

By Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, Hiroshi Saruwatari. 2022

634 citations recorded by the source corpus.

We present the UTokyo-SaruLab mean opinion score (MOS) prediction system submitted to VoiceMOS Challenge 2022. The challenge is to predict the MOS values of speech samples collected from previous Blizzard Challenges and Voice Conversion Challenges for two tracks: a main track for in-domain prediction and an out-of-domain (OOD) track for which there is less labeled data from different listening tests. Our system is based on ensemble learning of strong and weak learners. Strong learners incorporate several improvements to the previous fine-tuning models of self-supervised learning (SSL) models, while weak learners use basic machine-learning methods to predict scores from SSL features. In the Challenge, our system had the highest score on several metrics for both the main and OOD tracks. In addition, we conducted ablation studies to investigate the effectiveness of our proposed methods.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Tan
2. Yanqing Liu
3. Kai Shen
4. Zeqian Ju
5. Dongchao Yang
6. Shujie Liu
7. Sheng Zhao
8. Detai Xin

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Results are reported for both similarity to original prompt (SIM-O) and reconstructed prompt (SIM-R); 2) for speech quality, we employ UTMOS [59] which is a surrogate objective metric of MOS; 3) for Word Error Rate (WER), we use an ASR model 5 to transcribe generated speech.”
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, methodology): “Objective Evaluation For objective evaluation of our models, we employ the UTMOS (Saeki et al., 2022) automatic Mean Opinion Score (MOS) prediction system.”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, background): “• UTMOS [23], which is a powerful automatic speech quality assessment model that evaluates speech naturalness.”
• XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (2024, methodology): “For the objective evaluation, following [16] we have used the UTMOS model [32] to predict the Naturalness Mean Opin-ion Score (nMOS).”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “…metrics: naturalness mean opinion score (nMOS) and voice similarity MOS (sMOS) with a CI of 95%; and three objective metrics for naturalness: UTMOS [82], character error rate (CER), and word error rate (WER); two objective metrics for similarity: automatic speaker verification equal error rate…”