wav2vec: Unsupervised Pre-training for Speech Recognition

By Steffen Schneider, Alexei Baevski, Ronan Collobert, Michael Auli. 2019

Subfields: Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “The latent representations are fed to a Transformer network to build contextualized representations and the model is trained via a contrastive task where the true latent is to be distinguished from distractors [54, 49, 48, 28] (§ 2).”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, neutral): “Self-supervised learning of generic neural representations has gathered much recent interest with a large body of work in natural language processing (NLP) [1, 2], computer vision [3, 4] as well as speech processing [5, 6, 7].”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, neutral): “configurations, which is composed of seven 512-channel layers with strides [5,2,2,2,2,2,2] and kernel widths [10,3,3,3,3,2,2].”
Unsupervised Speech Recognition (2021, neutral): “Their model architecture and loss function was simplified in Schneider et al. (2019) who also applied it to full speech recognition.”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, positive): “In unsupervised pre-training, the knowledge from the pre-training task may transfer to the supervised task [21].”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, neutral): “Self-supervised learning of speech representations has received a lot attention [1, 2, 3, 4, 5] and has been demonstrated to work well both in low- and high-resource labeled data settings for automatic speech recognition (ASR) [6].”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, neutral): “…and speech translation (ST) has been achieved by the development and application of unsupervised speech pre-training methods (Oord et al., 2018; Schneider et al., 2019; Baevski et al., 2020; Conneau et al., 2020; Wu et al., 2020; Nguyen et al., 2020), with semi-supervised learning…”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Another line of work is pre-training representations on unlabeled speech followed by fine-tuning on labeled data [16, 17, 18, 19, 20, 21, 22, 23, 24].”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, positive): “For low-resource directions, we also evaluate self-supervised speech features (Schneider et al., 2019; Wu et al., 2020) 5 as an alternative to the traditional log mel-ﬁlter bank features (“+ SSL”).”
Improved Noisy Student Training for Automatic Speech Recognition (2020, neutral): “In this approach, a separate task/loss to (pre-)train the acoustic model (or a part thereof) is introduced to help the model learn a good representation of the input, e.g., [24, 25, 26, 27, 28, 29, 30].”
Learning Robust and Multilingual Speech Representations (2020, positive): “architectures, we use encoder and autoregressive models similar to (Schneider et al., 2019).”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, neutral): “In addition, a thorough comparison is needed with other recent work on representation learning approaches [8, 7, 6, 22] to separate the effects of model type versus pre-training approach; this is not trivial as it is not straightforward to extend these prior approaches to bidirectional recurrent mod”
Effectiveness of self-supervised pre-training for speech recognition (2019, positive): “Table 1: Librispeech WER of BERT models taking discretized or continuous input features: for discretized BERT quantized input units are obtained via vq-wav2vec or by k-means clustering of MFCC/FBANK features; for continuous BERT wav2vec features are learned following [12].”
VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, neutral): “For example, wav2vec [24] trains a multi-layer convolutional network optimized via a contrastive loss.”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, neutral): “wav2vec [38], vq-wav2vec [39], wav2vec 2.”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, neutral): “SSL methods have shown remarkable results for ASR [2, 3], phoneme segmentation [13], and GSLM [5].”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, neutral): “…more recently there is increasing interest in exploring speech pre-training using large quantities of unlabeled speech data (Chung et al., 2019; Schneider et al., 2019; Kharitonov et al., 2021; Baevski et al., 2020; Hsu et al., 2021c; Liu et al., 2020; Ling and Liu, 2020; Tjandra et al.,…”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, neutral): “In speech processing, most work in this area has focused on monolingual unsupervised representation learning [18, 19, 20, 21, 22, 23, 24, 25].”