End-to-end Feedback Loss in Speech Chain Framework via Straight-through Estimator

By Andros Tjandra, Sakriani Watiasri Sakti, Satoshi Nakamura. 2018

46 citations recorded by the source corpus.

The speech chain mechanism integrates automatic speech recognition (ASR) and text-to-speech synthesis (TTS) modules into a single cycle during training. In our previous work, we applied a speech chain mechanism as a semi-supervised learning. It provides the ability for ASR and TTS to assist each other when they receive unpaired data and let them infer the missing pair and optimize the model with reconstruction loss. If we only have speech without transcription, ASR generates the most likely transcription from the speech data, and then TTS uses the generated transcription to reconstruct the original speech features. However, in previous papers, we just limited our back-propagation to the closest module, which is the TTS part. One reason is that back-propagating the error through the ASR is challenging due to the output of the ASR being discrete tokens, creating non-differentiability between the TTS and ASR. In this paper, we address this problem and describe how to thoroughly train a speech chain end-to-end for reconstruction loss using a straight-through estimator (ST). Experimental results revealed that, with sampling from ST-Gumbel-Softmax, we were able to update ASR parameters and improve the ASR performances by 11% relative CER reduction compared to the baseline.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, background): “It can be done either by training a standalone multi-speaker TTS system on in-domain data [9] or by building and training of ASR and TTS systems jointly [10].”
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, background): “This can be regarded as a special case of speech chain with ST-estimator [13] where the ASR is a CTC network with temporal segmentation and the unpaired text is not utilized.”
VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, methodology): “[7, 8, 9] proposed a machine speech chain (see Figure 1) that enables ASR and TTS to assist each other when they receive unpaired data by allowing them to infer the missing pair and optimize both models with reconstruction loss.”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, background): “Another line of work adds a TTS model (Tjandra et al., 2017; 2019; Baskar et al., 2019) or a text-to-encoding (TTE) model (Hayashi et al., 2018; Hori et al., 2019) in the loop of ASR training, which can be utilized for back-translation style data augmentation (Sennrich et al., 2016) or…”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “This improves the TTE cycle loss in [9] and is related to the straight-Through loss in [8]”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “initially freezes either the ASR or TTS model, extensions of this study [303], [306], [307] implement the joint training of”
Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models (2019, methodology): “The approach of joint learning has been tested in the literature in other domains than NLG/NLU for tasks such machine translation (Cheng et al., 2016; He et al., 2016; Tu et al., 2017) and speech processing (Tjandra et al., 2017, 2018; Liu et al., 2018).”
Improving Speech Recognition Using Consistent Predictions on Synthesized Speech (2020, methodology): “We bridge the non-differentiable argmax on the ASR decoder using a Gumbel softmax as in [10].”
Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems (2019, methodology): “Another method is the joint training of ASR and text-to-speech (TTS) systems such as the Speech Chain approach [6–8] or variants of it [9].”