JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment

By D. Lim, Won Jang, Gyeonghwan O, Heayoung Park, Bongwan Kim, Jaesam Yoon. 2020

40 citations recorded by the source corpus.

We propose Jointly trained Duration Informed Transformer (JDI-T), a feed-forward Transformer with a duration predictor jointly trained without explicit alignments in order to generate an acoustic feature sequence from an input text. In this work, inspired by the recent success of the duration informed networks such as FastSpeech and DurIAN, we further simplify its sequential, two-stage training pipeline to a single-stage training. Specifically, we extract the phoneme duration from the autoregressive Transformer on the fly during the joint training instead of pretraining the autoregressive model and using it as a phoneme duration extractor. To our best knowledge, it is the first implementation to jointly train the feed-forward Transformer without relying on a pre-trained phoneme duration extractor in a single training pipeline. We evaluate the effectiveness of the proposed model on the publicly available Korean Single speaker Speech (KSS) dataset compared to the baseline text-to-speech (TTS) models trained by ESPnet-TTS.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Tan
2. Tao Qin
3. Yi Ren
4. Ye Jia
5. Tie-Yan Liu
6. Rongjie Huang
7. Yu Zhang
8. Chenpeng Du

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Tacotron [72] AM ph FF−→melS AR−→wav arXiv20 2020.10 DeviceTTS [123] AM ph AR−→Ceps→wav arXiv20 2020.10 Wave-Tacotron [379] E2E ch/ph AR−→wav ICASSP21 2020.11 DenoiSpeech [427] AM ph FF−→melS FF−→wav ICASSP21 2020.12 EfficientTTS [229] AM ch FF−→melS FF−→wav ICML21 2020.12 EfficientTTS-Wav [229] E2E”
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “Compared with previous non-autoregressive acoustic models [7, 11, 28], most of them focus on improving the duration accuracy to reduce the information gap between the input and output, while FastSpeech 2 and 2s take more missing information into account.”
• UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (2021, methodology): “Vocoders have been employed in various fields such as text-tospeech[1, 2, 3, 4], voice conversion[5], and speech-to-speech translation[6].”
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling (2020, methodology): “The WER is computed on speech synthesized on transcripts from the LibriTTS test-clean subset with the same 10 speakers in subsection 5.1, and then transcribed by an ASR model described in Park et al. (2020) with a WER of 2.3% on the ground truth audio.”
• Lightspeech: Lightweight and Fast Text to Speech with Neural Architecture Search (2021, methodology): “Non-autoregressiveTTS models [5, 9, 7, 8, 6] generate speech in parallel and greatly speed up the inference process compared to autoregressive TTS models [1, 2, 3].”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “To simplify the training process, JDI-T [121] jointly trains the autoregressive Transformer teacher network and the feed-forward Transformer student network.”
• Universal MelGAN: A Robust Neural Vocoder for High-Fidelity Waveform Generation in Multiple Domains (2020, unknown): “O, H. Park, B. Kim, and J. Yoon, “JDI-T: Jointly trained duration informed transformer for text-to-speech without explicit alignment,” arXiv preprint arXiv:2005.07799, 2020.”
• PortaSpeech: Portable and High-Quality Generative Text-to-Speech (2021, background): “Among neural network-based TTS systems, some of them generate mel-spectrograms autoregressively from text [15, 22, 29, 35] and suffer from slow inference speed and robustness (word skipping and repeating) problems [25], while others [8, 12, 16, 19, 24, 25] generate mel-spectrograms in parallel with ”
• JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech (2022, background): “Moreover, the training pipeline can be more simplified by incorporating alignment learning so that the required token duration is obtained during the model training on the fly [15], [18], [23], [24].”
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling (2021, methodology): “The proposed Parallel Tacotron 2 is a duration-based non-autoregressive neural TTS model which does not require supervised duration signals, like [18–21 AlignTTS [18], JDI-T [21], and EfficientTTS [20] train an alignment network jointly with a TTS model to produce alignments between tokens and frames.”
• Triple M: A Practical Text-to-Speech Synthesis System with Multi-Guidance Attention and Multi-Band Multi-Time LPCNet (2021, background): “Using a fixed diagonal mask to regulate the attention alignment matrix in training is helpful to solve this problem [15].”