Forward–Backward Decoding Sequence for Regularizing End-to-End TTS

By Yibin Zheng, Jianhua Tao, Zhengqi Wen, Jiangyan Yi. 2019

24 citations recorded by the source corpus.

Neural end-to-end TTS such as Tacotron like network can generate very high-quality synthesized speech, and even close to human recording for similar domain text. However, it performs unsatisfactory when scaling it to some challenging test sets. One concern is that the encoder-decoder with attention-based network adopts autoregressive generative sequence model with the limitation of “exposure bias”: errors made early could be quickly amplified, harming subsequent sequence generation. To address this issue, we propose two novel methods, which aim at predicting future by improving the agreement between forward and backward decoding sequence. The first one (denoted as MRBA) is achieved by adding divergence regularization terms to model training objective to maximize the agreement between two directional models, namely L2R (which generates targets from left-to-right) and R2L (which generates targets from right-to-left). While the second one (denoted as BDR) operates on decoder-level and exploits the future information during decoding. By introducing regularization term into the training objective of forward-backward decoders, the forward-decoder's hidden states are forced to be close to the backward-decoder's. Thus, the hidden representations of a unidirectional decoder are encouraged to embed some useful information about the future. Moreover, in order to make forward and backward decoding to improve each other in an interactive process, a joint training method is designed. Experimental results on both English and Mandarin dataset show that our proposed methods especially the second one (BDR), lead to a significantly improvement on both robustness and overall naturalness, as achieving obvious preference advantages in a challenging test, and achieving state-of-the-art performance (outperforming baseline “the revised version of Tacotron2” with a gap of 0.13 and 0.12 for English and Mandarin in MOS, respectively) on a general test.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Ye Jia
5. Frank K. Soong
6. Yu Zhang
7. Heiga Zen
8. Sheng Zhao

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Ye Jia
5. Frank K. Soong
6. Yu Zhang
7. Heiga Zen
8. Yonghui Wu

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Considering the right part of the generated mel-spectrogram sequence is usually worse than than that in the left part due to error propagation, some works leverage both left-to-right and right-to-left generations [338] for data augmentation [285] and regularization [445].”
• Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, background): “Together with the soft attention mechanism, it can lead to robustness errors such as babbling, early cut-off, word repetition, and word skipping [11–14].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[267] proposed two BSM methods for acoustic models, which take full advantage of the autoregressive model at the initial iteration stage and reduce errors in synthesized speech by adding bidirectional decoding regularization term to the loss function during training.”
• EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion (2021, background): “The most similar scheme may be the one in [16].”
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, background): “However, these types of models are known to be less robust than traditional approaches (He et al., 2019; Zheng et al., 2019; Guo et al., 2019; Battenberg et al., 2020).”
• Investigation of Fast and Efficient Methods for Multi-Speaker Modeling and Speaker Adaptation (2021, background): “Although the generated speech quality approaches human level, it needs a much larger amount of data to train such a high-quality textto-speech (TTS) model [1, 2, 3].”