Cycle-consistency Training for End-to-end Speech Recognition

By Takaaki Hori, Ramón Fernandez Astudillo, Tomoki Hayashi, Yu Zhang, Shinji Watanabe, Jonathan Le Roux. 2018

90 citations recorded by the source corpus.

This paper presents a method to train end-to-end automatic speech recognition (ASR) models using unpaired data. Although the end-to-end approach can eliminate the need for expert knowledge such as pronunciation dictionaries to build ASR systems, it still requires a large amount of paired data, i.e., speech utterances and their transcriptions. Cycle-consistency losses have been recently proposed as a way to mitigate the problem of limited paired data. These approaches compose a reverse operation with a given transformation, e.g., text-to-speech (TTS) with ASR, to build a loss that only requires unsupervised data, speech in this example. Applying cycle consistency to ASR models is not trivial since fundamental information, such as speaker traits, are lost in the intermediate text bottleneck. To solve this problem, this work presents a loss that is based on the speech encoder state sequence instead of the raw speech signal. This is achieved by training a Text-To-Encoder model and defining a loss based on the encoder reconstruction error. Experimental results on the LibriSpeech corpus show that the proposed cycle-consistency training reduced the word error rate by 14.7% from an initial model trained with 100-hour paired data, using an additional 360 hours of audio data without transcriptions. We also investigate the use of text-only data mainly for language modeling to further improve the performance in the unpaired data training scenario.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “Multiple methods were proposed to jointly train the ASR and TTS models in a cycle-consistent manner [22, 23, 24].”
Self-Training for End-to-End Speech Recognition (2019, background): “Prior work mainly Recently-proposed semi-supervised approaches for end-to-end speech recognition take advantage of text-to-speech (TTS) modules to generate synthetic data from unpaired text [22] or introduce a cycle-consistency loss between the input and the output of an ASR+TTS pipeline [9, 10].”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Compared to adversarial training (Liu et al., 2019), back-translation (Hayashi et al., 2018), and cycleconsistency (Hori et al., 2019), LPM achieves significantly better performance without the need to jointly train additional modules.”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “In Librispeech, the 100h set was used as paired data and the 360h set as unpaired data as in [9].”
Semi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text (2019, methodology): “For TTS, Tacotron2 [21] model is used and is configured as described in our previous work [9] differing only with the output targets.”
End-to-End Speech Recognition: A Survey (2023, unknown): “Furthermore,high-performanceE2Esolutionsexistfor bothdiscriminativeproblemslikeASR,aswellasgenerative problemslikeTTS,howcanbothbeexploitedjointlytosupport semi-supervisedtrainingbasedontext-onlyand/oraudio-only dataontopoftranscribedspeechaudio[28],[342]?”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, methodology): “Successful incorporation of unspoken text was demonstrated in a variety of tasks by connecting ASR and TTS via an end-to-end differentiable loss [1, 2, 4, 11].”
Improving Speech Recognition Using Consistent Predictions on Synthesized Speech (2020, methodology): “Cycle consistent loss was to applied to the SpeechChain setup [11, 15], where REIN-FORCE [16] and straight-through estimators were utilized to bridge the gradients between ASR decoding and TTS encoder.”
Machine Speech Chain (2020, background): “Hori et al. [36] replaced TTS with text-to-encoder (TTE) to avoid the need for modeling the speaking style during the reconstruction.”
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020, methodology): “In previous works, cycle-consistency based losses have been used as part of the training objective in machine translation (He et al., 2016) and speech recognition (Hori et al., 2019).”
Semi-supervised ASR by End-to-end Self-training (2020, methodology): “[18, 19, 20, 21, 22] leverage unpaired speech and text data by combining ASR with Text-to-Speech (TTS) modules, with a training loss that encourages pseudo-labels from ASR to reconstruct audio features well with the TTS system, and TTS outputs to be recognized by ASR.”
Rnn-Transducer with Stateless Prediction Network (2020, methodology): “Many recent works have investigated effective ways of using text-only data to improve E2E models, including fusion with an external LM [4, 5, 6, 7, 8], knowledge distillation from an LM [9], augmenting paired data with TTS data [10, 11], etc.”
An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, background): “Alternatively, “cycleconsistency” [17, 18] has also been explored to train E2E models on unpaired speech and text data.”
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, background): “w-resourcesetting,thesingle-speakerhigh-qualitypaired data are reduced to dozens of minutes in TTS [2, 12, 23, 31] while the multi-speaker low-quality paired data is reduced to dozens of hours in ASR [16, 32, 33, 39], compared to that in the richresource setting. Additionally, they leverage unpaired”
Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems (2019, methodology): “A previously published approach is the text-to-encoder (TTE) model which can integrate additional text [4] or untran-scribed speech [5] into ASR training.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, unknown): “…et al. (2019) propose to leverage speech synthesis systems to generate paired text and speech data for ASR training; Tjandra et al. (2017), Hori et al. (2019), and Baskar et al. (2019) chain ASR and TTS together to form a loop for semi-supervised learning with cycle-consistency loss;…”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, background): “[5, 6, 7, 8] (2) Augmenting audio data with contextual metadata as distant labels [9, 10, 11, 12].”
Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models (2020, methodology): “We start our formulation with unpaired speech data Y with the following expected loss based on the cycle consistency [15]: In our study, we simplify this training by using the pretrained ASR model (shown in Fig.”