An analysis of environment, microphone and data simulation mismatches in robust speech recognition

By Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, Ricard Marxer. 2017

401 citations recorded by the source corpus.

An analysis of the impact of acoustic mismatches between training and test data on the performance of robust ASR.Including: environment, microphone and data simulation mismatches.Based on: a critical analysis of the results published on the CHiME-3 dataset and new experiments.Result: with the exception of MVDR beamforming, these mismatches have little effect on the ASR performance.Contribution: the CHiME-4 challenge, which revisits the CHiME-3 dataset and reduces the number of microphones available for testing. Speech enhancement and automatic speech recognition (ASR) are most often evaluated in matched (or multi-condition) settings where the acoustic conditions of the training data match (or cover) those of the test data. Few studies have systematically assessed the impact of acoustic mismatches between training and test data, especially concerning recent speech enhancement and state-of-the-art ASR techniques. In this article, we study this issue in the context of the CHiME-3 dataset, which consists of sentences spoken by talkers situated in challenging noisy environments recorded using a 6-channel tablet based microphone array. We provide a critical analysis of the results published on this dataset for various signal enhancement, feature extraction, and ASR backend techniques and perform a number of new experiments in order to separately assess the impact of different noise environments, different numbers and positions of microphones, or simulated vs. real data on speech enhancement and ASR performance. We show that, with the exception of minimum variance distortionless response (MVDR) beamforming, most algorithms perform consistently on real and simulated data and can benefit from training on simulated data. We also find that training on different noise environments and different microphones barely affects the ASR performance, especially when several environments are present in the training data: only the number of microphones has a significant impact. Based on these results, we introduce the CHiME-4 Speech Separation and Recognition Challenge, which revisits the CHiME-3 dataset and makes it more challenging by reducing the number of microphones available for testing.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Speech & Audio

In Signal Processing

Citations in the corpus

CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (2020, background): “rocessing, audio enhancement, and machine learning, but also to the availability of real speech corpora recorded in cars [7,8], quiet indoor environments [9,10], noisy indoor and outdoor environments [11,12], and challenging broadcast media [13,14]. Among the applications of robust ASR, voice comman”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “With these state-of-theart end-to-end ASR techniques, ESPnet also provides a number of recipes for major ASR benchmarks including Wall Street Journal (WSJ) [18], Librispeech [19], TED-LIUM [20], Corpus of Spontaneous Japanese (CSJ) [21], AMI [22], HKUST Mandarin CTS [23], VoxForge [24], CHiME-4/5 [2”
Joint CTC-attention based end-to-end speech recognition using multi-task learning (2016, methodology): “We performed three sets of experiments: two on clean speech corpora, WSJ1 (81 hours) and WSJ0 (15 hours) [14, 15], and one on a noisy speech corpus, CHiME-4 (18 hours) [16].”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, methodology): “To verify our proposed method on a non-artificial dataset, we repeat our experiments on the CHiME-4 [17] dataset, which contains real distant-talking recordings in noisy environments.”
Student-teacher network learning with enhanced features (2017, methodology): “This paper experimentally demonstrates the effectiveness of the method by using the CHiME-4 speech separation and recognition challenge dataset [16], in which 1-, 2-, and 6-channel tracks are prepared by setting the subset of microphones that can be used in the test stage, while all 6 channel data c”
Speech Processing for Digital Home Assistants: Combining signal processing with deep-learning techniques (2019, background): “Notably, the Computational Hearing in Multisource Environments (CHiME) series of challenges [1], [2], the Reverberant Voice Enhancement and Recognition Benchmark (REVERB) Challenge [3], and the Automatic Speech Recognition in Reverberant Environments (ASpIRE) Challenge [4] were met with considerable”
A Study of Learning Based Beamforming Methods for Speech Recognition (2016, methodology): “We evaluate the three learning based beamforming methods on the 2-channel and 6-channel tracks of the CHiME-4 task [16].”
Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training (2020, methodology): “In our experiments, we use a multi-speaker Mandarin corpus, referred as MULTI-SPK, and a noise corpus from CHiME-4 challenge [25] to simulate noisy speech.”
The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation (2016, background): “In contrast to the provided baseline system [9] the back end has been replaced completely and is described in Section 2.2.”
Multi-Modal Data Augmentation for End-to-end ASR (2018, background): “Speech is also commonly corrupted with synthetic noise or reverberation [16, 17].”
Hybrid CTC/Attention Architecture for End-to-End Speech Recognition (2017, methodology): “The ﬁrst part of the experiments used famous English clean speech corpora, WSJ1 and WSJ0 [33], [34], and a noisy speech corpus, CHiME-4 [35].”
Conditional Diffusion Probabilistic Model for Speech Enhancement (2022, methodology): “While the scores on the CHiME4 test set are lower than the VoiceBank scores, CDiffuSE degrades to a much smaller degree than its regressive competitors, leaving it with the best scores on the CHiME-4 test data and demonstrating its high robustness to variation in noise characteristics.”
Front-end processing for the CHiME-5 dinner party scenario (2018, background): “Many of these developments have been spurred by recent challenges like REVERB [3], CHiME-3 [4] and CHiME-4 [5], which showcased the beneﬁts of a strong speech enhancement front-end for Automatic Speech Recognition (ASR).”
Speaker-Aware Neural Network Based Beamformer for Speaker Extraction in Speech Mixtures (2017, methodology): “In particular, a scheme combining neural network based mask estimation together with beamforming was suggested by different works [7, 8] and appears particularly effective [9, 10].”
Acoustic Modeling for Overlapping Speech Recognition: Jhu Chime-5 Challenge System (2019, background): “First, to tackle the distributed distant microphone setting, we explore data augmentation for improving robustness against a variety of room acoustics and directivity patterns of speech, which shows great success in noise robust ASR [7, 8, 9].”
Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization (2019, methodology): “These two 109 methods have been mainly applied for training robust discriminative models [9, 24, 26, 21], and 110 are less studied in the context of building generative models.”
Large-Scale Domain Adaptation via Teacher-Student Learning (2017, methodology): “In [14] it was shown that for the multi-channel CHiME-4 task [15], soft labels could be derived using enhanced features generated by a beamformer then processed through a network trained with conventional multi-style training [16].”
Voices Obscured in Complex Environmental Settings (VOICES) corpus (2018, methodology): “Integrated recording in real environments was introduced in later challenges[5, 6], but this included only 4 speakers in four different settings, recorded at close range microphony via 1, 2, or 6 microphones.”
An Investigation into the Effectiveness of Enhancement in ASR Training and Test for Chime-5 Dinner Party Transcription (2019, background): “However, there has been a long debate whether it is advisable to apply speech enhancement on data used for ASR training, because it is generally agreed upon that the recognizer should be exposed to as much acoustic variability as possible during training, as long as this variability matches the test”