BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

By Yu Zhang, Daniel S. Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang and 16 more. 2021

208 citations recorded by the source corpus.

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34 k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, methodology): “This pipelining approach has been used in BigSSL [41].”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “While not insignificant, this is still tiny compared to the previously mentioned 1,000,000 hours of unlabeled speech data utilized in Zhang et al. (2021).”
FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, methodology): “Speech technology has been rapidly evolved in the past few years, with striking achievements nourished from selfattention models [3, 4], pre-training approaches [5, 6, 7, 2], and massively multilingual speech models [8, 9].”
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “Target labels work BigSSL [47] also mentions large SSL model could handle various speech tasks.”
CLAP Learning Audio Concepts from Natural Language Supervision (2023, methodology): “The pretrained model is then adapted to a downstream task in a supervised setup learning under the class label paradigm [3, 4, 5].”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “Our work is an extension of a host of recent research efforts [3, 10, 13, 53, 55] that have studied semi-supervised learning for ASR in the context of deep-learning.”
Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, unknown): “recognition [55], and computer vision [54].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, unknown): “thousand hours supervised training data [289, 337].”
End-to-End Speech Recognition: A Survey (2023, unknown): “Pretraining approaches utilizing untranscribed audio, large-scale semi-supervised data and/or multilingual data [120], [121], [122], [123], [124], [125], [126], [127], [128], [129], [130], [131] would deserve”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “Furthermore, the utilization of a larger quantity of unlabeled data helps alleviate the overfitting issue, thereby further enhancing the overall performance of the BigSSL model (Zhang et al., 2022).”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, background): “The speech SSL models, such as Wav2vec 2.0 [Baevski et al., 2020], HuBERT [Hsu et al., 2021], BigSSL [Zhang et al., 2022], WavLM [Chen et al., 2022b], and data2vec [Baevski et al., 2022], show prominent performance across various speech processing tasks, especially in low-resource scenarios.”