Speech Recognition

By Steffen Schneider, Alexei Baevski, Ronan Collobert, Michael Auli. 2018

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

Citations in the corpus

Are we done with ImageNet? (2020, neutral): “Similarly, successful representation learning techniques applied to ImageNet [9, 10, 11] have yielded corresponding gains elsewhere [12, 13, 14, 15].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “It has been observed for Natural Language Processing [12, 39], Speech Recognition [43, 45] and Computer Vision [35].”
Unsupervised Pretraining Transfers Well Across Languages (2020, methodology): “This result not only conﬁrms the ﬁndings of [6] but it also shows that unsupervised pre-training can match supervised pre-training with enough data (see Supplementary Section S2 with the larger Libri-light dataset [29]).”
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, background): “So far, deep learning based speech representations were primary learned on frame level [3, 4, 5].”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “Recently, unsupervised pre-training has shown promising results in several areas, including Computer Vision (CV) [1], Natural Language Processing (NLP) [2, 3] and so on [4, 5, 6].”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, methodology): “Our new discretization algorithm, vq-wav2vec, learns discrete representations of ﬁxed length segments of audio signal by utilizing the wav2vec loss and architecture (Schneider et al, 2019; §2).”
Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning (2021, background): “Contrastive learning has recently attracted a surge of interest and shown promising results in various areas including computer vision [18, 6], natu-ral language processing [11, 15], audio processing [49, 42], graph [20, 44] and multimodal data [4, 51].”
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, neutral): “Prior works [36, 37, 38] utilize disentangled representation for speech generation, such as speech content from self-supervised pre-trained models [39, 40, 41], fundamental frequency, and timbre, but speech quality is not satisfying.”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Significant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are fine-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
AudioGen: Textually Guided Audio Generation (2022, methodology): “…phoneme segmentation (Kreuk et al., 2020), and audio compression (Zeghidour et al., 2021; Polyak et al., 2021). van den Oord et al. (2018) and Schneider et al. (2019) suggested training a convolutional neural network to distinguish true future samples from random distractor samples using a…”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, methodology): “Following previous works [2, 3, 4, 5, 6, 7, 8], we evaluate different features and representations on downstream tasks, including: phoneme classiﬁcation, speaker recognition, and sentiment classiﬁcation on spoken content.”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “The ﬁrst type is based on contrastive loss [17] and has been applied to speech representation such as wav2vec and its variants [14, 18, 15].”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “Recently, self-supervised learning—a paradigm that treats the input itself or modiﬁcations of the input as learning targets—has obtained promising results for learning such transformations [1, 2, 4, 5, 6, 7, 8, 9, 3, 10].”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, background): “While speech representations can be defined by different transformations on the surface feature, recent researches [1, 2, 3, 4, 5, 6, 7, 8, 9] have shown great success by combining neural networks and self-supervised learning (where learning targets can be derived from the input itself).”
Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation (2022, unknown): “Examples include contrastive predictive coding (CPC) [16, 29], auto-regressive predictive coding [30], wav2vec [31], HuBERT [32,33], wav2vec 2.”
Wav2CLIP: Learning Robust Audio Representations from Clip (2021, unknown): “Self-supervised learning is a highly active and fruitful area of research, yielding state-of-the-art (SOTA) results in speech recognition [1, 2, 3], music classiﬁcation [4, 5], computer vision [6, 7], and natural language processing [8, 9].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “The wav2vec model [121] extends the CPC approach and uses fully convolutional parameterizations for the modules f1(·) and f2(·) with receptive fields of 30ms and 210ms, respectively.”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “Prediction-based self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions [29, 30, 31, 32, 33, 34, 35] or to contrast the target unseen frame with randomly sampled ones [5, 36, 6, 11].”
Variable Bitrate Discrete Neural Representations via Causal Self-Attention (2021, background): “Learning compressed discrete codes from continuous domain data has been shown to enable efﬁcient latent generative modeling [24, 5, 23, 27, 26].”
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, methodology): “Self-supervised pre-training for speech In speech, wav2vec (Schneider et al., 2019) leverages contrastive learning to produce contextual representations for audio input; vq-wav2vec (Baevski et al.”