Listening while speaking: Speech chain by deep learning

By Andros Tjandra, Sakriani Watiasri Sakti, Satoshi Nakamura. 2017

175 citations recorded by the source corpus.

Despite the close relationship between speech perception and production, research in automatic speech recognition (ASR) and text-to-speech synthesis (TTS) has progressed more or less independently without exerting much mutual influence on each other. In human communication, on the other hand, a closed-loop speech chain mechanism with auditory feedback from the speaker's mouth to her ear is crucial. In this paper, we take a step further and develop a closed-loop speech chain model based on deep learning. The sequence-to-sequence model in close-loop architecture allows us to train our model on the concatenation of both labeled and unlabeled data. While ASR transcribes the unlabeled speech features, TTS attempts to reconstruct the original speech waveform based on the text from ASR. In the opposite direction, ASR also attempts to reconstruct the original text transcription given the synthesized speech. To the best of our knowledge, this is the first deep learning model that integrates human speech perception and production behaviors. Our experimental results show that the proposed approach significantly improved the performance more than separate systems that were only trained with labeled data.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, methodology): “3 showed our model (part (c)) was more robust for generating intelligible speech for long input text than models without codebook (part (b)) while Speech Chain (part (c)) could hardly ﬁnish most of the sentences.”
VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, methodology): “Recently, Tjan-dra et al. [7, 8, 9] proposed a machine speech chain (see Figure 1) that enables ASR and TTS to assist each other when they receive unpaired data by allowing them to infer the missing pair and optimize both models with reconstruction loss.”
A Survey on Neural Speech Synthesis (2021, methodology): “Techniques like speech chain [344, 345] and back transformation [285, 390] leverage additional unpaired text and speech data to boost the performance of TTS and ASR.”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “Below we provide a brief overview of the self-supervised audio representations that are essential for SPEAR-TTS.”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “Furthermore, it is can useful to construct a systematic method for synthesizing speech with random latent attributes, which would facilitate data augmentation (Tjandra et al., 2017; 2018; Hsu et al., 2017b; 2018; Hayashi et al., 2018) by generating diverse examples.”
The Zero Resource Speech Challenge 2019: TTS without T (2019, background): “Recent research shows that training ASR and TTS jointly with reconstruction losses can result in improvement in both [15].”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Another line of work adds a TTS model (Tjandra et al., 2017; 2019; Baskar et al., 2019) or a text-to-encoding (TTE) model (Hayashi et al.”
Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, background): “Some papers [30, 31, 32] also show the performance improvement from the synthetic dataset can be carried over to the real dataset.”
Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, methodology): “…Wang et al., 2017; Shen et al., 2018; Ping et al., 2019) and ASR (Chiu et al., 2018; Zhang et al., 2018; Xiong et al., 2018; Yang et al., 2018) (Tjandra et al., 2017; 2018) relied on two well-trained ASR and TTS models to further boost the accuracy with unpaired speech and text data, which is…”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “This improves TTE backtranslation [10] and is related to the speech-chain in [13]”
Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models (2019, methodology): “The approach of joint learning has been tested in the literature in other domains than NLG/NLU for tasks such machine translation (Cheng et al., 2016; He et al., 2016; Tu et al., 2017) and speech processing (Tjandra et al., 2017, 2018; Liu et al., 2018).”
End-to-End Speech Recognition: A Survey (2023, background): “…is how to take advantage of text-only and audio-only data jointly without introducing secondary (pretrained) models and/or training objectives [28], [29]. e) Direct Vocabulary Modeling: Avoiding pronunciation lex-ica and corresponding subword units leave E2E recognition vocabularies to be…”
Injecting Text in Self-Supervised Speech Pretraining (2021, background): “Training methodologies to jointly learn from unpaired speech and text such as Deep Chain [14], cycle-consistency training [15], and augmentation approaches [16, 17] are becoming increasingly popular.”
Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model (2018, background): “Another approach is to back-translate (synthesize) speech signals or encoder state sequences [17, 21, 22] from the unpaired text data, so they can be jointly learned in training.”
DUB: Discrete Unit Back-translation for Speech Translation (2023, background): “Moreover, the discrete unit extractor is obtained through unsupervised training on unlabeled speech, which requires no transcriptions compared with the ASR system trained on speech-transcription pairs.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “There are some cycle-consistency works using unpaired speech-text data to improve ASR accuracy [16, 274].”
Semi-Supervised Sequence-to-Sequence ASR Using Unpaired Speech and Text (2019, methodology): “Examples of this are the speech-chain framework [13] and adversarial training schemes [14, 15].”
Semi-Supervision in ASR: Sequential MixMatch and Factorized TTS-Based Augmentation (2021, methodology): “Over the last few years, text-to-speech (TTS) synthesis-based data augmentation has been shown to be a useful way to train end-to-end models using unspoken text [1–4].”
Cycle-consistency Training for End-to-end Speech Recognition (2018, methodology): “Previous approaches related to cycle consistency in end-toend ASR [9, 12] circumvent these problems by avoiding backpropagating the error beyond the discrete steps and adding a speaker network to transfer the information not present in the text.”