Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training

By Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve and 1 more. 2021

279 citations recorded by the source corpus.

Self-supervised learning of speech representations has been a very active research area but most work is focused on a single domain such as read audio books for which there exist large quantities of labeled and unlabeled data. In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain. Our experiments show that using target domain data during pre-training leads to large performance improvements across a variety of setups. On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%. This has obvious practical implications since it is much easier to obtain unlabeled target domain data than labeled data. Moreover, we find that pre-training on multiple domains improves generalization performance on domains not seen during training. Code and models will be made available at https://github.com/pytorch/fairseq.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

In Speech & Audio

In Signal Processing

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, neutral): “…et al., 2020) • facebook/wav2vec2-base-960h (Baevski et al., 2020) • facebook/wav2vec2-large-960h (Baevski et al., 2020) • facebook/hubert-large-ls960-ft (Hsu et al., 2021a) • facebook/hubert-xlarge-ls960-ft (Hsu et al., 2021a) • facebook/s2t-medium-librispeech-asr (Wang et al., 2020a) •…”
Unsupervised Speech Recognition (2021, methodology): “Hsu et al. (2021b) leverages automatically discovered speech units to learn representations with a BERT-like masked prediction objective and achieves comparable results with models based on contrastive learning.”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “[14] trains wav2vec 2.0 [5] on larger and more diverse datasets, but there are still over 90 % audio data derived from audiobook.”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, methodology): “…is increasing interest in exploring speech pre-training using large quantities of unlabeled speech data (Chung et al., 2019; Schneider et al., 2019; Kharitonov et al., 2021; Baevski et al., 2020; Hsu et al., 2021c; Liu et al., 2020; Ling and Liu, 2020; Tjandra et al., 2020; Hsu et al., 2021b,a).”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, methodology): “In this work, we focus our analysis on the open-source wav2vec 2.0 (W2V2) models [11], which have been successful for speech recognition [14–16] and translation [17].”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, unknown): “w2v2-b [18] LibriSpeech 960 eng R hubert-b [19] LibriSpeech 960 eng R w2v2-L [18] Libri-Light 60k eng R hubert-L [19] Libri-Light 60k eng R w2v2-L-robust [38] Libri-Light (60k) Fisher (2k) CommonVoice (700) Switchboard (300) 63k eng R, T”
Self-Supervised Speech Representation Learning: A Review (2022, background): “per, there is already some related research about enhancing the robustness of SSL models [281], [352], [353], [354] and identifying their vulnerability to adversarial attack [351].”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “Indeed, W2V2 has already been studied with focus on the impact of pretraining data (Conneau et al., 2020; Hsu et al., 2021), pre-training task (Hsu et al.”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “Recent successful representation learning frameworks for speech (e.g., APC (Chung et al., 2019), CPC (Oord et al., 2018; Kharitonov et al., 2021), wav2vec 2.0 (Baevski et al., 2020; Hsu et al., 2021b), De-CoAR2.”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “In this section, we turn to building multilingual ASR models on data from multiple domains following a similar approach as prior work for English-only models [Likhomanenko et al., 2020, Chan et al., 2021, Hsu et al., 2021].”
Injecting Text in Self-Supervised Speech Pretraining (2021, background): “Previous work [38] analyzed the impact of the domain of untranscribed speech on self-supervised pretraining with the conclusion that in-domain untranscribed speech is more valuable that outdomain speech.”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, background): “, 2021) on multiple domains (Hsu et al., 2021) and languages (Conneau et al.”