Effectiveness of self-supervised pre-training for speech recognition

By Alexei Baevski, Michael Auli, Abdel-rahman Mohamed. 2019

158 citations recorded by the source corpus.

We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model. We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features. Fine-tuning a BERT model on 10 hour of labeled Librispeech data with a vq-wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on testclean, while achieving a 25% WER reduction on test-other. When using only 10 minutes of labeled data, WER is 25.2 on test-other and 16.3 on test-clean. This demonstrates that self-supervision can enable speech recognition systems trained on a near-zero amount of transcribed data.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “Instead of fixed positional embeddings which encode absolute positional information, we use a convolutional layer similar to [37, 4, 57] which acts as relative positional embedding.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “In addition, we include a comparable result from DiscreteBERT [51] in Table VII which applies k-”
Unsupervised Speech Recognition (2021, methodology): “Wav2vec-U, or wav2vec Unsupervised, leverages self-supervised representations from wav2vec 2.0 (Baevski et al., 2020c) to embed the speech audio and to segment the ∗.”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, methodology): “We experiment with wav2vec 2.0 [6, 19] which consists of a convolutional feature encoder f : X (cid:55)→ Z to map raw audio x to latent speech representations z 1 , . . . , z T input to a Transformer g : Z (cid:55)→ C to output context representations c 1 , . . . , c T [20, 21].”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “This model contains a convolutional feature encoder f : X (cid:55)→ Z to map raw audio X to latent speech representations z 1 , . . . , z T which are input to a Transformer g : Z (cid:55)→ C to output context representations c 1 , . . . , c T [26, 18, 27].”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “Many algorithms exist which incorporate unlabelled data to improve ASR in the low-resource setting, including CPC [7, 8], pseudo-labeling [9], local prior matching [10], pseudo-label augmentation [11], adversarial training [12] and back translation [13].”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, neutral): “A recent paper [27] obtains impressive results with CPC, VQ and BERT pretraining on 960h of unlabelled data and fine-tuning on train-10h (test-clean: 10.”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We compare with several state-of-the-art self-supervised approaches, including DiscreteBERT (Baevski et al., 2019), wav2vec 2.0 (Baevski et al., 2020), and HuBERT (Hsu et al., 2021).”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “…et al., 2020; Chi et al., 2021; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski et al., 2019; Hsu et al., 2021), derived signal processing features (Pascual et al., 2019; Ravanelli et al., 2020), and binary labels of…”
Textless Speech-to-Speech Translation on Real Data (2021, positive): “Following the success of self-supervised pre-training and Connec-tionist Temporal Classiﬁcation (CTC) ﬁnetuning for ASR (Graves et al., 2006; Baevski et al., 2019), we propose to build a speech normalizer by performing CTC ﬁnetuning with a pre-trained speech encoder using multi-speaker speech as…”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “In vq-wav2vec and DiscreteBERT, a problematic token ID assignment would negatively affect the subsequent learning module and it is hard for the learning module to recover the errors made by the discretizer.”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “The discrete version of BERT + vq-wav2vec [5] uses a two-step pre-”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “Recently, self-supervised learning—a paradigm that treats the input itself or modiﬁcations of the input as learning targets—has obtained promising results for learning such transformations [1, 2, 4, 5, 6, 7, 8, 9, 3, 10].”
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020, background): “…such as audio waveforms or spectrograms, to higher-level representations using only unlabeled speech, has received great attention recently (Baevski et al., 2019, 2020; Liu et al., 2020; Song et al., 2019; Jiang et al., 2019; Schneider et al., 2019; Chorowski et al., 2019; Pascual et al.,…”
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training (2023, unknown): “To address this challenge, pre-trained language models (PLMs) for acoustic music have been proposed to provide reusable learned representations, enabling transfer learning for various downstream MIR tasks without the need for extensive data annotation (Castellon et al., 2021).”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “(2020a) introduced vq-wav2vec and a combination of vq-wav2vec with a discrete BERT-like model (Devlin et al., 2019; Baevski et al., 2019).”
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS (2023, methodology): “Baevski et al. [11] examined the utilization of discrete tokens in automatic speech recognition (ASR) and found underperformed compared to traditional FBank features unless combined with a BERT [12] as a preprocessing step.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “The Discrete BERT approach [129] uses a pre-trained vq-wav2vec model to derive a discrete vocabulary [124].”