Back-Translation-Style Data Augmentation for end-to-end ASR

By Tomoki Hayashi, Shinji Watanabe, Yu Zhang, Tomoki Toda, Takaaki Hori, Ramón Fernandez Astudillo, Kazuya Takeda. 2018

109 citations recorded by the source corpus.

In this paper we propose a novel data augmentation method for attention-based end-to-end automatic speech recognition (E2E-ASR), utilizing a large amount of text which is not paired with speech signals. Inspired by the back-translation technique proposed in the field of machine translation, we build a neural text-to-encoder model which predicts a sequence of hidden states extracted by a pre-trained E2E-ASR encoder from a sequence of characters. By using hidden states as a target instead of acoustic features, it is possible to achieve faster attention learning and reduce computational cost, thanks to sub-sampling in E2E-ASR encoder, also the use of the hidden states can avoid to model speaker dependencies unlike acoustic features. After training, the text-to-encoder model generates the hidden states from a large amount of unpaired text, then E2E-ASR decoder is retrained using the generated hidden states as additional training data. Experimental evaluation using LibriSpeech dataset demonstrates that our proposed method achieves improvement of ASR performance and reduces the number of unknown words without the need for paired data.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

Unsupervised Speech Recognition (2021, methodology): “Other work explored data augmentation via a technique similar to back-translation (Hayashi et al., 2018) as used in machine translation (Bojar and Tamchyna, 2011; Sennrich et al., 2015; Edunov et al., 2018).”
Self-Training for End-to-End Speech Recognition (2019, background): “Prior work mainly Recently-proposed semi-supervised approaches for end-to-end speech recognition take advantage of text-to-speech (TTS) modules to generate synthetic data from unpaired text [22] or introduce a cycle-consistency loss between the input and the output of an ASR+TTS pipeline [9, 10].”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “self-training [18, 29] or back-translation [30] from unlabeled data have been shown to be useful to ASR or S2T.”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “Furthermore, it is can useful to construct a systematic method for synthesizing speech with random latent attributes, which would facilitate data augmentation (Tjandra et al., 2017; 2018; Hsu et al., 2017b; 2018; Hayashi et al., 2018) by generating diverse examples.”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “, 2019), back-translation (Hayashi et al., 2018), and cycleconsistency (Hori et al.”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “This improves TTE backtranslation [10] and is related to the speech-chain in [13]”
Semi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text (2019, methodology): “0% relative improvement over backtranslation-TTE [10] approach.”
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, methodology): “Unlike previous ideas for leveraging translation data [20, 21, 22], our approach does not require any modification to the ASR model architecture.”
DUB: Discrete Unit Back-translation for Speech Translation (2023, background): “A similar data augmentation idea through synthesizing speech from utterances can be applied to automatic speech recognition (ASR) (Tjandra et al., 2017; Hayashi et al., 2018; Ueno et al., 2021).”
Improving Speech Recognition Using Consistent Predictions on Synthesized Speech (2020, background): “Another extension draws the TTS predictions further upstream to ASR encoder states [14].”
Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model (2018, methodology): “Another approach is to back-translate (synthesize) speech signals or encoder state sequences [17, 21, 22] from the unpaired text data, so they can be jointly learned in training.”
Cycle-consistency Training for End-to-end Speech Recognition (2018, methodology): “We call this a text-to-encoder (TTE) model [8], which we introduced in our prior work on data augmentation.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “For example, Hayashi et al. (2018) and Rosenberg et al. (2019) propose to leverage speech synthesis systems to generate paired text and speech data for ASR training; Tjandra et al. (2017), Hori et al. (2019), and Baskar et al. (2019) chain ASR and TTS together to form a loop for semi-supervised…”
Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit (2019, methodology): “For example, ASRbased objective evaluation for TTS systems and advanced research topics such as the semi-supervised learning [27]–[30] can be realized by combining ASR and TTS modules in the unified framework.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “, encoder features [319] and phoneme features [320], [321].”
End-to-End Speech Recognition: A Survey (2023, methodology): “Also, text-only data may be used to generate data using textto-speech (TTS) on feature [178] or signal level [179].”
Injecting Text in Self-Supervised Speech Pretraining (2021, background): “Leveraging vast amounts of unpaired text through learned text representations have been explored using shared encoder representations in [18, 19, 20].”
Data Augmentation for ASR Using TTS Via a Discrete Representation (2021, unknown): “Another direction that has been pursued is incorporation of unpaired text data into (part of) the ASR model training [8, 9, 10, 11, 12].”
Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems (2019, methodology): “A previously published approach is the text-to-encoder (TTE) model which can integrate additional text [4] or untranscribed speech [5] into ASR training.”