Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text

By Murali Karthick Baskar, Shinji Watanabe, Ramón Fernandez Astudillo, Takaaki Hori, Lukáš Burget, Jan Honza Cernocký. 2019

37 citations recorded by the source corpus.

Sequence-to-sequence ASR models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for self-supervised and supervised training in such models. This work builds upon recent results showing notable improvements in self-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with text-to-speech (TTS) models. In particular, this work proposes a new self-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with a point estimate TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided to reproduce the experiments.

Subfields: Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “Multiple methods were proposed to jointly train the ASR and TTS models in a cycle-consistent manner [22, 23, 24].”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “Many algorithms exist which incorporate unlabelled data to improve ASR in the low-resource setting, including CPC [7, 8], pseudo-labeling [9], local prior matching [10], pseudo-label augmentation [11], adversarial training [12] and back translation [13].”
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition (2019, background): “While semi-supervised learning has been exploited in a plethora of works in hybrid ASR system, there are very few work done in the end-to-end counterparts [4, 14, 15].”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “6 Cycle-TTS (Baskar et al., 2019) 100hr 360hr (S) None - - 17.”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, methodology): “• computer vision: graph-based methods [25, 26], generative modelling [27, 28], consistency-based methods [29, 30, 31, 32], and contrastive methods [33, 34, 35, 36]; • machine translation (MT): integration of a language model trained on monolingual data [37, 38, 39], backtranslation [40, 41, 42], sy”
Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit (2019, methodology): “For example, we provide the recipe based on ASR-TTS cycle consistency training [29] and semi-supervised training using ASR and TTS [28].”
Machine Speech Chain (2020, methodology): “[35] proposed an alternative to backpropagate through discrete variables by using a policy-gradient method, compared to our proposal using a straight-through estimator.”
Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation (2020, background): “The pretraining model learns the robust speech representations for speech processing tasks, for example, ASR and speaker recognition, with the self-supervised learning approaches [4, 5, 6, 7, 8].”
Improving Speech Recognition Using Consistent Predictions on Synthesized Speech (2020, methodology): “Cycle consistent loss was to applied to the SpeechChain setup [11, 15], where REINFORCE [16] and straight-through estimators were utilized to bridge the gradients between ASR decoding and TTS encoder.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “…to leverage speech synthesis systems to generate paired text and speech data for ASR training; Tjandra et al. (2017), Hori et al. (2019), and Baskar et al. (2019) chain ASR and TTS together to form a loop for semi-supervised learning with cycle-consistency loss; Weiss et al. (2017), Li et…”
Semi-supervised ASR by End-to-end Self-training (2020, methodology): “[18, 19, 20, 21, 22] leverage unpaired speech and text data by combining ASR with Text-to-Speech (TTS) modules, with a training loss that encourages pseudo-labels from ASR to reconstruct audio features well with the TTS system, and TTS outputs to be recognized by ASR.”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, background): “Therefore, purely supervised training is impractical for the vast majority of the 7,000 languages spoken around the world [5] which is why there has been a lot of interest in how to better use unlabeled speech data [6, 7, 8].”
Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems (2019, methodology): “We performed our simulated low-resource experiments on LibriSpeech [26], similar to previous work (e.g. [9]).”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “There are some cycle-consistency works using unpaired speech-text data to improve ASR accuracy [16, 274].”
Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation (2021, background): “MSE loss can be interpreted as the minus log-probability of the speech features for Gaussian and Laplace distributions −logPTTS(X|Y ) for constant scale parameters [22]:”