wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
By Alexei Baevski, Henry Zhou, Abdel-rahman Mohamed, Michael Auli. Neural Information Processing Systems, 2020
8,824 citations recorded by the source corpus.
We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.
Subfields: Artificial Intelligence, Speech & Audio
View Paper: https://arxiv.org/abs/2006.11477
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Kyunghyun Cho
In Speech & Audio
- Jinyu Li
- Wei-Ning Hsu
- Yu Zhang
- James R. Glass
- Shinji Watanabe
- Abdel-rahman Mohamed
- Yossi Adi
- Gabriel Synnaeve
Citations in the corpus
- Evaluating Large Language Models Trained on Code (2021, background): “…Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019;…”
- Scaling Laws for Acoustic Models (2021, methodology): “Successful techniques use either autoregressive predictive coding (APC) [7, 8, 9] or contrastive predictive coding (CPC) [10, 11, 12] to pre-train an acoustic model from a large corpus of unlabeled speech.”
- Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “…(Xu et al., 2021) • facebook/wav2vec2-large-robust-ft-libri-960h (Hsu et al., 2021b) • facebook/wav2vec2-base-100h (Baevski et al., 2020) • facebook/wav2vec2-base-960h (Baevski et al., 2020) • facebook/wav2vec2-large-960h (Baevski et al., 2020) •…”
- MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “The speech encoder θs is trained on speech input using a within-utterance contrastive loss focusing on local acoustic and phonetic information [3, 9].”
- mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Given the convergence of architectures (Vaswani et al., 2017) and objectives (Devlin et al., 2019; Baevski et al., 2020; Chung et al., 2021) across the speech and text modalities, building a single model that could learn cross-lingual cross-modal representations of speech and text from hundreds of…”
- XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “For speech processing, self-supervised approaches such as wav2vec 2.0 [7, 12] have also been extended to the multilingual setting [13, 14].”
- Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, neutral): “The discriminative meth-ods usually employ contrastive learning [5] or classification on weak clustering label [6] to pre-train an encoder network with large-scale unsupervised data.”
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “creasing interests recently, where a model is tasked to predict the content of the unseen regions [4], [44]–[50] or to contrast the target unseen frame with randomly sampled ones [1]– [3], [6].”
- Unsupervised Speech Recognition (2021, methodology): “Wav2vec-U, or wav2vec Unsupervised, leverages self-supervised representations from wav2vec 2.0 (Baevski et al., 2020c) to embed the speech audio and to segment the ∗.”
- SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, positive): “There has also been prior work on pretraining on unlabelled audio, and finetuning on supervised data [21, 15].”
- Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, methodology): “It is well known that the performance of ASR systems trained from scratch with conventional supervised objectives can degrade significantly when tested on domains mismatched from training data [7, 8].”
- VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “0 (Baevski et al., 2020) substantially reduces the need of labelled data in ASR.”
- Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Recently, substantial performance improvements have been made by leveraging audio-only data for speech recognition [14, 15, 16].”
- Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “8% on the clean and other test sets of Librispeech, a relative WER reduction of 25% and 40% over recent work in pre-training alone [24].”
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “0 pre-training [6] to pre-train the encoders of the networks.”