SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

By William Chan, Daniel S. Park, Chris Lee, Yu Zhang, Quoc V. Le, Mohammad Norouzi. 2021

152 citations recorded by the source corpus.

We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Speech & Audio

In Signal Processing

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “As demonstrated by Narayanan et al. (2018), Likhomanenko et al. (2020), and Chan et al. (2021) speech recognition systems that are pre-trained in a supervised fashion across many datasets/domains exhibit higher robustness and generalize much more effectively to held-out datasets than models trained…”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “Comparing Maestro with TTS4Pretrain 2.0, the text encoder here is learnt from all the SpeechStew data while TTS model used in TTS4Pretrain 2.0 is only trained on Librispeech.”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “We then present our results on two public tasks, SpeechStew [30] and CHiME-6 [70] and compare them to the current SoTA benchmarks.”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “main source, but the massive audiobook data mismatches the data in a real scenario and using it exclusively hurts the model performance when the acoustic characteristics of the downstream tasks are different from those of the audiobook [10]–[13].”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “We combine S2T datasets from multiple domains to improve the robustness of model training [51, 52], resulting in 196-hr training data for Es-En and 571-hr for En-Es.”
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, background): “Recently it was shown that Transformer [48] architectures are state-of-the-art on different public benchmarks for ASR [55, 33, 30, 8].”
Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone (2021, background): “[28] proposed to mix various corpora to pre-train a single large ASR, showing a WER of 21.”
Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “This yields an additional 3.5M utterances (over the 40M from Librispeech) from the SpeechStew [56] training set, a combination of 7 publicly available supervised speech corpora.”
MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud (2022, background): “There is a growing body of research showing that large Deep Learning (DL) models deliver superior accuracy in areas such as natural language processing (NLP) [10, 46], speech recognition (SR) [6, 8, 54], and computer vision (CV) [9, 52, 53].”
SpeakerStew: Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Verification System (2021, background): “Evidence from speech recognition also indicates combining more training data improves speech models in general [30].”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “In this section, we turn to building multilingual ASR models on data from multiple domains following a similar approach as prior work for English-only models [Likhomanenko et al., 2020, Chan et al., 2021, Hsu et al., 2021].”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, background): “Following Chan et al. (2021a), the whole dataset (approx.”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “This includes SpeechStew (mono-lingual ASR) [2], CORAAL (African American Vernacular English (AAVE) ASR) [17], FLEURS (multi-lingual ASR) [16], YT (multilingual long-form ASR), and CoVoST (AST from English to multiple languages).”