The third 'CHiME' speech separation and recognition challenge: Analysis and outcomes

By Jon Barker, Ricard Marxer, Emmanuel Vincent, Shinji Watanabe. 2017

141 citations recorded by the source corpus.

This paper presents the design and outcomes of the CHiME-3 challenge, the first open speech recognition evaluation designed to target the increasingly relevant multichannel, mobile-device speech recognition scenario. The paper serves two purposes. First, it provides a definitive reference for the challenge, including full descriptions of the task design, data capture and baseline systems along with a description and evaluation of the 26 systems that were submitted. The best systems re-engineered every stage of the baseline resulting in reductions in word error rate from 33.4% to as low as 5.8%. By comparing across systems, techniques that are essential for strong performance are identified. Second, the paper considers the problem of drawing conclusions from evaluations that use speech directly recorded in noisy environments. The degree of challenge presented by the resulting material is hard to control and hard to fully characterise. We attempt to dissect the various 'axes of difficulty' by correlating various estimated signal properties with typical system performance on a per session and per utterance basis. We find strong evidence of a dependence on signal-to-noise ratio and channel quality. Systems are less sensitive to variations in the degree of speaker motion. The paper concludes by discussing the outcomes of CHiME-3 in relation to the design of future mobile speech recognition evaluations.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Speech & Audio

In Signal Processing

Citations in the corpus

CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (2020, neutral): “This can be attributed to advances in speech processing, audio enhancement, and machine learning, but also to the availability of real speech corpora recorded in cars [7, 8], quiet indoor environments [9, 10], noisy indoor and outdoor environments [11, 12], and challenging broadcast media [13,14].”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, background): “For example, we applied speed perturbation [27] at ratio 0.9, 1.0 and 1.1 to CSJ, CHiME4, Fisher-CALLHOME Spanish, HKUST, and TED-LIUM2/3, and we also applied SpecAugment [26] to Aurora4, LibriSpeech, TED-LIUM2/3 and WSJ.2”
Multi-Channel Deep Clustering: Discriminative Spectral and Spatial Embeddings for Speaker-Independent Speech Separation (2018, background): “Their beamforming approaches follow the success of T-F masking and deep learning based beamforming for speech enhancement [12] in the recent CHiME-3 and CHiME-4 challenges [13], where the estimated single-channel masks are combined to compute speech or noise covariance matrices.”
An analysis of environment, microphone and data simulation mismatches in robust speech recognition (2017, unknown): “Our analysis diﬀers from the one of Barker et al. (2016), which focuses on the speaker characteristics and the noise characteristics of each environment and compares the achieved ASR performance with the intelligibility predicted using perceptual models.”
The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines (2018, unknown): “This can be attributed to advances in speech processing, audio enhancement, and machine learning, but also to the availability of real speech corpora recorded in cars [6, 7], quiet indoor environments [8, 9], noisy indoor and outdoor environments [10, 11], and challenging broadcast media [12,13].”
Voices Obscured in Complex Environmental Settings (VOICES) corpus (2018, methodology): “Integrated recording in real environments was introduced in later challenges[5, 6], but this included only 4 speakers in four different settings, recorded at close range microphony via 1, 2, or 6 microphones.”