End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures

By Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, Ronan Collobert. 2019

264 citations recorded by the source corpus.

We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

End-to-End Object Detection with Transformers (2020, neutral): “replacing RNNs in many problems in natural language processing, speech processing and computer vision [7,26,44,33,30].”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “Finally, we also mention the concurrent work of Synnaeve et al. (2019), where they also apply a convolutional Trans-former architecture similiar to ours, showing strong empirical results on both CTC and seq2seq models.”
Self-training Improves Pre-training for Natural Language Understanding (2020, background): “, 2020), automatic speech recognition (Synnaeve et al., 2019; Kahn et al., 2020; Park et al., 2020), sequence generation (He et al.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “The Transformer LM is identical to [51] and contains 20 blocks, model dimension 1,280, inner dimension 6,144 and 16 attention heads.”
Unsupervised Speech Recognition (2021, methodology): “We phonemize the Librispeech LM corpus, with Librispeech and Librivox text data removed (Synnaeve et al., 2020).”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “Multi-lingual training [10, 11], multi-domain training [12, 13], unsupervised pre-training [14, 15], semi-supervised learning [16, 17] and transfer learning [18, 19] are some techniques proposed in the literature to enhance generalization.”
MLS: A Large-Scale Multilingual Dataset for Speech Research (2020, methodology): “Following [19], we also apply a channel increasing factor F = 2 in each TDS block.”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “(2020; [2]): first, we generate a list of candidate transcriptions by combining wav2vec 2.”
Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, methodology): “All models are trained with Connectionist Temporal Classification [25] and the network architecture follows [26]: the encoder of our acoustic models is composed of a convolutional frontend (1-D convolution with kernel-width 7 and stride 3 followed by GLU activation) followed by 36 4-heads Transforme”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “To alleviate overﬁt-ting, we pre-train ST model encoders on English ASR and adopt SpecAugment (without time warping): LD policy on LibriSpeech models and LB policy on MuST-C and CoVoST 2 models.”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “More recently, as end-to-end models have matured in the monolingual setting [10, 11], attention has turned to leverage multiple languages to further improve their performance.”
Iterative Pseudo-Labeling for Speech Recognition (2020, positive): “A simple extension to [1] involves conducting more iterations of PL as the model trains so as to continuously refine and improve the quality of generated pseudolabels.”
You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, background): “Semi-supervised learning is one of the external data usage techniques proven to be effective [5].”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, neutral): “Our medium model, ContextNet(M), only has 31M parameters and achieves similar WER compared with much larger systems [11, 33].”
Learning Robust and Multilingual Speech Representations (2020, positive): “Although better results contain be obtained using newer architectures than TDNN (Park et al., 2019; Synnaeve et al., 2019), it still represents a standard and important recognition architecture and the results prove that the representations learned from diverse and noisy data can improve large speec”