Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model

By Alexander H. Liu, Hung-yi Lee, Lin-Shan Lee. 2018

47 citations recorded by the source corpus.

In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM). In this way the CLM and the automatic speech recognition (ASR) model can challenge and learn from each other iteratively to improve the performance. Since the CLM only takes the text as input, huge quantities of unpaired text data can be utilized in this approach within end-to-end training. Moreover, AT can be applied to any end-to-end ASR model using any deep-learning-based language modeling frameworks, and compatible with any existing end-to-end decoding method. Initial results with an example experimental setup demonstrated the proposed approach is able to gain consistent improvements efficiently from auxiliary text data under different scenarios.

Subfields: Artificial Intelligence, Large Language Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Speech & Audio

Citations in the corpus

Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Therefore, purely supervised training is impractical for the vast majority of the 7,000 languages spoken around the world [5] which is why there has been a lot of interest in how to better use unlabeled speech data [6, 7, 8].”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “Adversarial training [12] LS-100 LS-860 (text) NN 15.”
Self-Training for End-to-End Speech Recognition (2019, background): “briSpeech. All numbers are reportedwithout an externalLM. A. APPENDIX A.1. Supervised Baseline Acommonsetupforsemi-supervisedASRistousethe“trainclean-100”subset of LibriSpeechas the labelled data set [22, 24]. Table 3 shows the WER fromour supervisedbaseline on “train-clean-100”as well as several ot”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Liu et al. (2019) treats ASR as a generative model that conditions on speech instead of random noise vectors, and adopts the generative adversarial network (GAN) (Goodfellow et al., 2014) framework in order to improve the fidelity of ASR-generated texts.”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “9% were observed by integrating RNNLM with this approach and the result is compared with crictizing-LM [12].”
Semi-supervised ASR by End-to-end Self-training (2020, methodology): “On the other hand, [24] use adversarial training to encourage ASR output on unsupervised data to have similar distribution as that of unpaired text data, with a criticizing language model.”
Semi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text (2019, methodology): “9% were observed by integrating RNNLM with this approach and the result is compared with crictizing-LM [12].”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, background): “40, 41, 42], synthetic data usage [43], and web-scale bitext mining [44]; • automatic speech recognition (ASR): representation learning [45, 2, 46, 47], local prior matching [4], adversarial training [48], back-translation [49] and others [50, 51, 52, 53, 54]. Below, we give an overview of the appro”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, unknown): “The most popular framework for this combination is the Deep Chain framework [1], with adversarial [17] and cycle consistency training objectives [18] providing increased robustness for ASR.”
Injecting Text in Self-Supervised Speech Pretraining (2021, background): “Adversarial [23] and cycle consistency training objectives [24] have also been proposed to leverage unpaired data.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “It becomes the standard training recipe for most AED models [52, 95, 121, 127, 166, 277, 284, 300].”
SpeechNet: A Universal Modularized Model for Speech Processing Tasks (2021, methodology): “(13) The objective function is similar to those used in previous works (Karita et al., 2019; Liu et al., 2019), which is a weighted sum of a S2S loss and a CTC loss: LASR = −α logPCTC(Y′CTC”