CROWDMOS: An approach for crowdsourcing mean opinion score studies

By Flavio P. Ribeiro, Dinei A. F. Florêncio, Cha Zhang, M. Seltzer. 2011

280 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

On Fast Sampling of Diffusion Probabilistic Models (2021, methodology): “, 2016), and the crowdMOS tookit (Ribeiro et al., 2011) to evaluate generated samples.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, methodology): “Results: We use the crowdMOS tookit (Ribeiro et al., 2011) for speech quality evaluation, where the test utterances from all models were presented to Mechanical Turk workers.”
Deep Voice: Real-time Neural Text-to-Speech (2017, methodology): “To estimate perceptual quality of the individual stages of our TTS pipeline, we crowdsourced mean opinion score (MOS) ratings (ratings between one and ﬁve, higher values being better) from Mechanical Turk using the CrowdMOS toolkit and methodology (Ribeiro et al., 2011).”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “Prior studies often adopt subjective metrics like mean opinion scores (MOS) [52] which are not comparable across papers, or quantitative metrics like mel cepstral distortion [35] that assume the output is deterministic given input, which is often not realistic [54].”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, methodology): “In this paper, we evaluate synthesized speech quality using a Mean Opinion Score (MOS) study, as in [42].”
On Generative Spoken Language Modeling from Raw Audio (2021, positive): “The CrowdMOS package (Ribeiro et al., 2011) was used for all subjective experiments using the recommended recipes for detecting and discarding inaccurate scores.”
An Open source Implementation of ITU-T Recommendation P.808 with Validation (2020, background): “CrowdMOS [6] ACR, MUSHRA Y N N Y Y (N=2) QualityCrowd2 [7] ACR Y N N N N BeaqleJS [8] ABX, MUSHRA Y N N N N WESP+ [9] ACR, DCR, PC, Y N N N N DSIS, DSCQS webMUSHRA [10] AB, ABX, Y N N N N MUSHRA, ACR CAQE [11] MUSHRA, PC N Y N Y N P.”
TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese (2020, methodology): “According to [48], the obtained value indicates a good quality audio, with a barely perceptible, but not annoying, distortion.”
Fitting New Speakers Based on a Short Untranscribed Sample (2018, methodology): “The MOS measure is obtained using the crowdMOS toolkit by (P. Ribeiro et al., 2011) and Amazon Mechanical Turk (AMT).”
Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017, methodology): “We use the crowdMOS toolkit (Ribeiro et al., 2011); batches of samples from these models were presented to raters on Mechanical Turk.”
Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, methodology): “We run an MOS evaluation using the crowdMOS framework (Ribeiro et al., 2011) to compare the quality of samples (Table 1).”
ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech (2018, methodology): “We use the crowdMOS toolkit (Ribeiro et al., 2011), where batches of samples from these models were presented to workers on Mechanical Turk.”
Generative Spoken Dialogue Language Modeling (2022, methodology): “The CrowdMOS package (Ribeiro et al., 2011) was used for all subjective evaluations using the recommended recipes for detecting and discarding inaccurate scores.”
Real Time Speech Enhancement in the Waveform Domain (2020, methodology): “For that, we launched a crowd source evaluation using the CrowdMOS package [28].”
AudioGen: Textually Guided Audio Generation (2022, methodology): “We use the CrowdMOS package 6 to ﬁlter noisy annotations and outliers.”
WaveFlow: A Compact Flow-based Model for Raw Audio (2019, methodology): “We use the crowdMOS tookit (Ribeiro et al., 2011) for speech quality evaluation, where test utterances from these models were presented to workers on Mechanical Turk.”
SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model (2021, methodology): “In this paper, the synthesized speech quality is evaluated using mean opinion score (MOS) study, following [32].”
Bilateral Denoising Diffusion Models (2021, positive): “We refer to the MOS scores in [46], and the scoring criteria have been included in Table 4 for completeness.”