Contrastive Semi-Supervised Learning for ASR

By Alex Xiao, Christian Fuegen, Abdel-rahman Mohamed. 2021

21 citations recorded by the source corpus.

Pseudo-labeling is the most adopted method for pre-training automatic speech recognition (ASR) models. However, its performance suffers with degrading quality of the supervised teacher model.Inspired by the successes of contrastive representation learning for both computer vision and speech applications, and more recently for supervised learning of visual objects [1], we propose Contrastive Semi-supervised Learning (CSL). CSL eschews directly predicting teacher generated pseudo-labels in favor of utilizing them to select positive and negative examples.In the challenging task of transcribing public social media videos, using CSL reduces the WER by 8%, compared to the standard Cross-Entropy pseudo-labeling (CE-PL), when 10hr of supervised data is used to annotate 75,000hr of videos. The WER reduction jumps to 19% under the ultra low-resource condition of using 1hr labels for teacher supervision. In out-of-domain conditions, CSL generalizes much better showing up to 17% WER reduction compared to the strongest CE-PL pre-trained model.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “Next, a student model is trained using the combined supervised and teacher-labeled data either using the standard cross-entropy [9] loss or using a contrastive loss [11] to account for noise in teacher-generated labels.”