Effectiveness of Self-Supervised Pre-Training for ASR

By Alexei Baevski, Abdel-rahman Mohamed. 2020

98 citations recorded by the source corpus.

We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] self-supervision approach to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model. We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features. Fine-tuning a BERT model on 10 hour of labeled Librispeech data with a vq-wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on test-clean, while achieving a 25% WER reduction on test-other. When using only 10 minutes of labeled data, WER is 25.2 on test-other and 16.3 on test-clean. This demonstrates that self-supervision can enable speech recognition systems trained on a near-zero amount of transcribed data.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “, 2020; Ling and Liu, 2020), cluster indices (Baevski and Mohamed, 2020; Hsu et al., 2020b), derived signal processing features (Pascual et al.”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “Also, the discrete BERT + vq-wav2vec [4] is built by stacking a standard BERT model [32] of 12 Transformer Encoder layers [28] on top of vq-wav2vec [3],”
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training (2023, methodology): “It is worth noting that our model utilises a convolutional relative positional embedding, similar to the approach introduced by Baevski et al. (2020) in Wav2Vec, enabling it to operate effectively in longer contexts, if required.”
Utilizing Self-supervised Representations for MOS Prediction (2021, background): “Self-supervised learning has achieved remarkable performance in several tasks including automatic speech recognition [16], speaker verification [17], voice conversion [18], and so on.”
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning (2022, methodology): “In consideration with a similar observation in [9], we hypothesize that BERT-based representation learning may be more effective in discrete than in continuous signal spaces.”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, methodology): “Prior work in self-supervised speech recognition has begun to synthesize parts of CPC and MLM to produce methodologies for self-learning with raw waveforms [13, 44, 45, 41, 31].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “The Discrete BERT approach [129] uses a pre-trained vq-wav2vec”