Semi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text

By Murali Karthick Baskar, Shinji Watanabe, Ramón Fernandez Astudillo, Takaaki Hori, Lukáš Burget, Jan Honza Cernocký. 2019

30 citations recorded by the source corpus.

Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models. This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models. In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided in ESPnet to reproduce the experiments.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Therefore, purely supervised training is impractical for the vast majority of the 7,000 languages spoken around the world [5] which is why there has been a lot of interest in how to better use unlabeled speech data [6, 7, 8].”
End-to-End Speech Recognition: A Survey (2023, methodology): “Also, attention-based encoder-decoder models have been shown to suffer from low training resources [222], [223], which can be improved by a number of approaches, including regularization techniques [174] as well as data augmentation using SpecAug-ment [224] and text-to-speech (TTS) [29].”
Injecting Text in Self-Supervised Speech Pretraining (2021, background): “Adversarial [23] and cycle consistency training objectives [24] have also been proposed to leverage unpaired data.”
Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models (2020, methodology): “Also, joint-training methods of end-to-end ASR and TTS have been studied, e.g., machine speech chain [13,14], cycle consistency [15,16], and voice conversion [17].”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, unknown): “The most popular framework for this combination is the Deep Chain framework [1], with adversarial [17] and cycle consistency training objectives [18] providing increased robustness for ASR.”