Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously
By Cheng-Han Chiang, Wei Huang, Hunghuei Lee. 2023
26 citations recorded by the source corpus.
This paper emphasizes the importance of reporting experiment details in subjective evaluations and demonstrates how such details can significantly impact evaluation results in the field of speech synthesis. Through an analysis of 80 papers presented at INTERSPEECH 2022, we find a lack of thorough reporting on critical details such as evaluator recruitment and filtering, instructions and payments, and the geographic and linguistic backgrounds of evaluators. To illustrate the effect of these details on evaluation outcomes, we conducted mean opinion score (MOS) tests on three well-known TTS systems under different evaluation settings and we obtain at least three distinct rankings of TTS models. We urge the community to report experiment details in subjective evaluations to improve the reliability and interpretability of experimental results.
Subfields: Developmental and Educational Psychology, Management Science and Operations Research, Sociology and Political Science
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Developmental and Educational Psychology
In Management Science and Operations Research
Citations in the corpus
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “Furthermore, recent work shows the necessity for disclosing the details of evaluation procedures for TTS research [43].”
- StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “A similar discrepancy has been reported in a very recent study that examines the effects of evaluation methods on the MOS results [62], and our evaluation of VITS is comparable to other studies that have tried to reproduce it on both LJSpeech and LibriTTS datasets [63], [64], [65], [66].”