Machine Speech Chain with One-shot Speaker Adaptation

By Andros Tjandra, Sakriani Watiasri Sakti, Satoshi Nakamura. 2018

58 citations recorded by the source corpus.

In previous work, we developed a closed-loop speech chain model based on deep learning, in which the architecture enabled the automatic speech recognition (ASR) and text-to-speech synthesis (TTS) components to mutually improve their performance. This was accomplished by the two parts teaching each other using both labeled and unlabeled data. This approach could significantly improve model performance within a single-speaker speech dataset, but only a slight increase could be gained in multi-speaker tasks. Furthermore, the model is still unable to handle unseen speakers. In this paper, we present a new speech chain mechanism by integrating a speaker recognition model inside the loop. We also propose extending the capability of TTS to handle unseen speakers by implementing one-shot speaker adaptation. This enables TTS to mimic voice characteristics from one speaker to another with only a one-shot speaker sample, even from a text without any speaker information. In the speech chain loop mechanism, ASR also benefits from the ability to further learn an arbitrary speaker's characteristics from the generated speech waveform, resulting in a significant improvement in the recognition rate.

Subfields: Artificial Intelligence, Speech & Audio, Meta & Continual Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Jinyu Li
5. Yu Zhang
6. Shinji Watanabe
7. Dong Yu
8. Hung-yi Lee

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Yu Zhang
4. Shinji Watanabe
5. Hung-yi Lee
6. Tao Qin
7. Tie-Yan Liu
8. Sheng Zhao

In Meta & Continual Learning

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Yang Chen
5. Weiran Wang
6. Chao Wang
7. Yu Zhang
8. Satoshi Nakamura

In Signal Processing

1. Jinyu Li
2. Shinji Watanabe
3. Xu Tan
4. Yu Zhang
5. Tao Qin
6. Tie-Yan Liu
7. Dong Yu
8. Hung-yi Lee

Citations in the corpus

• VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, neutral): “[7, 8, 9] proposed a machine speech chain that enables ASR and TTS to assist each other when they receive unpaired data by allowing them to infer the missing pair and optimize both models with reconstruction loss.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Techniques like speech chain [344, 345] and back transformation [285, 390] leverage additional unpaired text and speech data to boost the performance of TTS and ASR.”
• Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “It is also desirable to construct a systematic method for synthesizing speech with random latent attributes, which can facilitate data augmentation (Tjandra et al., 2017; 2018; Hsu et al., 2017b; 2018; Hayashi et al., 2018) by providing more diverse examples.”
• Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, methodology): “…Chen et al., 2018b), low-resource ASR (Chuang-suwanich, 2016; Dalmia et al., 2018; Zhou et al., 2018), TTS with minimal speaker data (Chen et al., 2019; Jia et al., 2018; Arik et al., 2018; Wang et al., 2018), and boosting TTS and ASR simultaneously in a speech chain (Tjandra et al., 2017; 2018).”
• Generalizing from a Few Examples (2019, background): “Recently, it is possible to perform voice conversion from one user to another using one-shot voice or text sample [128] or even across different languages [93].”
• Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation (2021, background): “Similar idea has been proposed in [21], which uses one-shot speaker adaptation to leverage more unseen speakers, to further improve both TTS and ASR.”
• Semi-supervised ASR by End-to-end Self-training (2020, methodology): “[18, 19, 20, 21, 22] leverage unpaired speech and text data by combining ASR with Text-to-Speech (TTS) modules, with a training loss that encourages pseudo-labels from ASR to reconstruct audio features well with the TTS system, and TTS outputs to be recognized by ASR.”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “Speech synthesis helps to reduce word error rate (WER) in customization of automatic speech recognition (ASR) especially for low resource or OOV scenarios by generating synthetic speech using text-to-speech (TTS), and using the text and speech pairs to update the ASR models or use a spelling correct”
• Semi-Supervised Speaker Adaptation for End-to-End Speech Synthesis with Pretrained Models (2020, methodology): “[14,25], which is also trained with paired data and x-vector [26].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, background): “It is suitable for various speech processing tasks, such as voice conversion, multi-speaker TTS and ASR.”
• Cycle-consistency Training for End-to-end Speech Recognition (2018, background): “Previous approaches related to cycle consistency in end-toend ASR [9, 12] circumvent these problems by avoiding backpropagating the error beyond the discrete steps and adding a speaker network to transfer the information not present in the text.”
• Self-Supervised Speech Representation Learning: A Review (2022, methodology): “A speech chain [302], [305] is one successful way to utilize audio-only and text-only data to train both end-to-end ASR/TTS models.”
• Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation (2018, methodology): “[17] builds a cycle chain between TTS and ASR models, in which the output from one model is used to help the training of the other.”
• Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “This is similar to the use of TTS in ASR training as in [20, 34], auxiliary ASR losses are also used in encoder training for voice conversion in [35].”
• End-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator (2018, methodology): “In this work, our core architecture is based on Tacotron [12] with several structural modifications [3].”
• Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems (2019, methodology): “Another method is the joint training of ASR and text-to-speech (TTS) systems such as the Speech Chain approach [6–8] or variants of it [9].”