Joint Masked CPC And CTC Training For ASR

By Chaitanya Talnikar, Tatiana Likhomanenko, Ronan Collobert, Gabriel Synnaeve. 2020

30 citations recorded by the source corpus.

Self-supervised learning (SSL) has shown promise in learning representations of audio that are useful for automatic speech recognition (ASR). But, training SSL models like wav2vec 2.0 requires a two-stage pipeline. In this paper we demonstrate a single-stage training of ASR models that can utilize both unlabeled and labeled data. During training, we alternately minimize two losses: an unsupervised masked Contrastive Predictive Coding (CPC) loss and the supervised audio-to-text alignment loss Connectionist Temporal Classification (CTC). We show that this joint training method directly optimizes performance for the downstream ASR task using unsupervised data while achieving similar word error rates to wav2vec 2.0 on the Librispeech 100-hours dataset. Finally, we postulate that solving the contrastive task is a regularization for the supervised CTC loss.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “When fine-tuning wav2vec 2.0 models, we follow Baevski et al. (2020) to add a linear projection layer on top and follow Talnikar et al. (2020) to optimize jointly on supervised CTC loss and unsupervised wav2vec loss.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “2020; Khurana, Laurent, and Glass 2021) and proven well-suited for concurrently learning with labeled data (Talnikar et al. 2021; Wang et al. 2021).”
Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “Pretraining techniques such as MoCo [4], Contrastive Predicting Coding (CPC) [5, 6], Autoregressive Per-dictive Coding (APC) [7], SimCLR [8], etc., generalize using un-transcribed speech, however, they cannot leverage unspoken text, thereby limiting the power of the learned representations.”
Joint Unsupervised and Supervised Training for Multilingual ASR (2021, background): “[21] alternatively minimizes an unsupervised masked CPC loss and a supervised CTC loss [22].”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, background): “One of the known problems with contrastive training is a need of large batches [28, 30].”