w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

By Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, Yonghui Wu. 2021

560 citations recorded by the source corpus.

Motivated by the success of masked language modeling (MLM) in pre-training natural language processing models, we propose w2v-BERT that explores MLM for self-supervised speech representation learning. w2v-BERT is a framework that combines contrastive learning and MLM, where the former trains the model to discretize input continuous speech signals into a finite set of discriminative speech tokens, and the latter trains the model to learn contextualized speech representations via solving a masked prediction task consuming the discretized tokens. In contrast to existing MLM-based speech pre-training frameworks such as HuBERT, which relies on an iterative re-clustering and re-training process, or vq-wav2vec, which concatenates two separately trained modules, w2v-BERT can be optimized in an end-to-end fashion by solving the two self-supervised tasks (the contrastive task and MLM) simultaneously. Our experiments show that w2v-BERT achieves competitive results compared to current state-of-the-art pre-trained models on the LibriSpeech benchmarks when using the Libri-Light 60k corpus as the unsupervised data. In particular, when compared to published models such as conformer-based wav2vec 2.0 and HuBERT, our model shows 5% to 10% relative WER reduction on the test-clean and test-other subsets. When applied to the Google's Voice Search traffic dataset, w2v-BERT outperforms our internal conformer-based wav2vec 2.0 by more than 30% relatively.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, methodology): “In this paper, we provide baselines for the tasks based on multilingual pre-trained models like speech-only w2vBERT [1] and speech-text multimodal mSLAM [2].”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “Similar to W2v-BERT [9], we use contrastive loss on the speech encoder outputs and a masked language model (MLM) loss on the shared encoder output to pretrain our model on untranscribed speech data.”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “Fine-tuning Setup: We fine-tune our pre-trained encoders with a 2-layer LSTM (Hochreiter & Schmidhuber, 1997) as a conformer-transducer model, following Chung et al. (2021).”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “Multilingually pretrained NLP models such as mBERT [2], XLM-R [8] or mT5 [9] brought significant improvements in multilingual language understanding [10, 11].”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, positive): “Prior works [36, 37, 38] utilize disentangled representation for speech generation, such as speech content from self-supervised pre-trained models [39, 40, 41], fundamental frequency, and timbre, but speech quality is not satisfying.”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “Anyone who feels that if so many more students whom we haven’t actually admitted are sitting in on the course than ones we have that the room had to be changed, then probably auditors will have to be excluded, is likely to agree that the curriculum needs revision.”
RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “Discrete speech tokens can be divided into two categories: Semantic tokens (Lee et al., 2022b; Hsu et al., 2021; Chung et al., 2021) and Acoustic tokens (Zeghidour et al., 2022; Défossez et al., 2022; Wu et al., 2023).”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, positive): “[2022] extract embeddings from the w2v-BERT model [Chung et al., 2021] and quantize them via k-means.”
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Textless NLP (Lakhotia et al. 2021) and AudioLM (Borsos et al. 2022) propose to leverage wav2vec 2.0 and w2v-BERT (Chung et al. 2021) respectively for language model training and consequently are able to generate speech unconditionally via auto-regressive inference.”
PolyVoice: Language Models for Speech to Speech Translation (2023, background): “Semantic units are usually derived from representations produced by speech encoder models like HuBERT (Hsu et al., 2021), mHuBERT (Lee et al., 2021) or w2v-BERT (Chung et al., 2021).”
SoundStorm: Efficient Parallel Audio Generation (2023, positive): “Unsuper-vised speech embeddings (Baevski et al., 2020; Hsu et al., 2021; Chung et al., 2021) have provided a low-framerate representation of the underlying signal which remains rich enough after discretization for language models to generate intelligible speech from a speciﬁc speaker as a sequence…”
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “In AudioLM, the input audio is mapped to semantic tokens using w2v-BERT [Chung et al., 2021] and acoustic tokens using SoundStream [Zeghidour et al., 2021].”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “• We compare the semantic tokens extracted from a pretrained w2v-BERT [17] and the acoustic tokens from SoundStream [16] on a speech dataset, and we show that they complement each other in terms of phonetic discriminability and reconstruction quality.”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, neutral): “0 [5], DiscreteBERT [40], HuBERT [6] and w2v-BERT [41].”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Although some speech-language pre-training work (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021) attempts to improve spoken language understanding tasks, these methods only focus on an encoder with task-speciﬁc layers for different tasks and do not pre-train a decoder for generation tasks…”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, background): “Acoustic tokens Acoustic tokens are discrete audio representations that provide high-ﬁdelity reconstruction of the acoustic details.”
MusicLM: Generating Music From Text (2023, methodology): “In particular, by following the approach of AudioLM, we use the self-supervised audio representations of SoundStream (Zeghidour et al., 2022), as acoustic tokens to enable high-ﬁdelity synthesis, and w2v-BERT (Chung et al., 2021), as semantic tokens to facilitate long-term coherent generation.”
Measuring Forgetting of Memorized Training Examples (2022, background): “, 2018), speech (Chung et al., 2021), and vision (Radford et al.”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “2Quantization is optional for wav2vec 2.0 (Baevski et al., 2020b; Zhang et al., 2020) but helpful for noisy speech (Chung et al., 2021).”
Are Discrete Units Necessary for Spoken Language Modeling? (2022, methodology): “Large-scale self-supervised pre-training for speech data has also become more and more popular as a method to boost the performance of Automatic Speech Recognition (ASR) [6]– [8].”