Almost Unsupervised Text to Speech and Automatic Speech Recognition

By Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu. 2019

111 citations recorded by the source corpus.

Text to speech (TTS) and automatic speech recognition (ASR) are two dual tasks in speech processing and both achieve impressive performance thanks to the recent advance in deep learning and large amount of aligned speech and text data. However, the lack of aligned data poses a major practical problem for TTS and ASR on low-resource languages. In this paper, by leveraging the dual nature of the two tasks, we propose an almost unsupervised learning method that only leverages few hundreds of paired data and extra unpaired data for TTS and ASR. Our method consists of the following components: (1) a denoising auto-encoder, which reconstructs speech and text sequences respectively to develop the capability of language modeling both in speech and text domain; (2) dual transformation, where the TTS model transforms the text $y$ into speech $\hat{x}$, and the ASR model leverages the transformed pair $(\hat{x},y)$ for training, and vice versa, to boost the accuracy of the two tasks; (3) bidirectional sequence modeling, which addresses error propagation especially in the long speech and text sequence when training with few paired data; (4) a unified model structure, which combines all the above components for TTS and ASR based on Transformer model. Our method achieves 99.84% in terms of word level intelligible rate and 2.68 MOS for TTS, and 11.7% PER for ASR on LJSpeech dataset, by leveraging only 200 paired speech and text data (about 20 minutes audio), together with extra unpaired speech and text data.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Jinyu Li
5. Yu Zhang
6. Frank K. Soong
7. Shinji Watanabe
8. Lei He

In Speech & Audio

1. Xu Tan
2. Yu Zhang
3. Jinyu Li
4. Tao Qin
5. Hung-yi Lee
6. Tie-Yan Liu
7. Sheng Zhao
8. Shinji Watanabe

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Jinyu Li
5. Frank K. Soong
6. Yu Zhang
7. Hung-yi Lee
8. Shinji Watanabe

Citations in the corpus

• MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “For future work, we will apply MASS to more language generation tasks such as sentence paraphrasing, text style transfer and post editing, as well as other sequence generation tasks (Ren et al., 2019).”
• Multitask Training with Text Data for End-to-End Speech Recognition (2020, methodology): “Multiple methods were proposed to jointly train the ASR and TTS models in a cycle-consistent manner [22, 23, 24].”
• Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, methodology): “We followed the prior work [14] to randomly choose the development/test set with 300 audio clips in each set and a different amount of paired data (5/15/20 minutes) from the remaining data.”
• Semi-Supervised Neural Architecture Search (2020, methodology): “We measure the performances in terms of word level intelligibility rate (IR), which is a commonly used metric to evaluate the quality of generated audio [22].”
• A Survey on Neural Speech Synthesis (2021, methodology): “Enhancing AR Professor forcing [96, 201] Reducing training/inference gap [355] Knowledge distillation [204] Bidirectional regularization [285, 445] Replacing AR with NAR Parallel generation [284, 286, 262, 67]”
• UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, neutral): “Following the common practice in low-resource and unsupervised speech and translation works [18, 31, 32], we conduct experiments on popular written languages but remove the text of target speech to simulate unwritten languages.”
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, background): “sion aims togenerate a sequence of pronunciation symbols (phonemes) given a sequence of letters (graphemes), which is an important component in automatic speech recognition and text-to-speech systems [1, 2] to provide accurate pronunciations for the words not covered by the lexicon. G2P conversion c”
• Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit (2019, methodology): “For example, ASRbased objective evaluation for TTS systems and advanced research topics such as the semi-supervised learning [27]–[30] can be realized by combining ASR and TTS modules in the unified framework.”
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, background): “[19] proposed to jointly train a phoneme recognition model and a speech synthesis model with unpaired data.”
• Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, background): “These approaches have shown great promise in ASR when combined with both transcribed speech (paired) and untranscribed speech (unpaired) [3, 16].”
• Data-Efficient Training Strategies for Neural TTS Systems (2020, background): “Another work [18] also aims for unsupervised text-to-speech models by obtaining supervision directly from an ASR model.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Librispeech, made up of audiobooks from the LibriVox project, was originally designed for ASR research and contains 982 hours of speech data from 2,484 speakers.”
• Injecting Text in Self-Supervised Speech Pretraining (2021, background): “These approaches have shown to be effective for ASR when combined with both transcribed and untranscribed speech [21, 22].”
• Semi-supervised ASR by End-to-end Self-training (2020, methodology): “[18, 19, 20, 21, 22] leverage unpaired speech and text data by combining ASR with Text-to-Speech (TTS) modules, with a training loss that encourages pseudo-labels from ASR to reconstruct audio features well with the TTS system, and TTS outputs to be recognized by ASR.”
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, methodology): “As a result, [31] cannot synthesize reasonable speech in TTS and achieves high WER according to our preliminary experiments.”
• Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, background): “…with discrete latents trained on unpaired target speeches (Zhang & Lin, 2020; Liu et al., 2020a), or dual learning between recognition and synthesis, along with denoising autoencoding (Ren et al., 2019), and then added by rich-resource pretraining and knowledge distillation (Xu et al., 2020).”
• Machine Speech Chain (2020, methodology): “, [31] replaced the LSTMbased encoder-decoder with Transformer modules for both ASR and TTS and achieved good performance with small paired speech-text in single speaker dataset.”
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks (2021, methodology): “It is called the length regulation technique originally proposed in TTS (Ren et al., 2019a).”
• Improving Speech Recognition Using Consistent Predictions on Synthesized Speech (2020, background): “An alternative approach to bridge the representation gap is by sharing encoders and decoders between TTS and ASR [17, 18].”
• Efficient Bidirectional Neural Machine Translation (2019, background): “The same phenomenon was also observed in Liu et al. (2016a); Wu et al. (2018); Ren et al. (2019).”