DurIAN: Duration Informed Attention Network For Multimodal Synthesis

By Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei and 2 more. 2019

119 citations recorded by the source corpus.

In this paper, we present a generic and robust multimodal synthesis system that produces highly natural speech and facial expression simultaneously. The key component of this system is the Duration Informed Attention Network (DurIAN), an autoregressive model in which the alignments between the input text and the output acoustic features are inferred from a duration model. This is different from the end-to-end attention mechanism used, and accounts for various unavoidable artifacts, in existing end-to-end speech synthesis systems such as Tacotron. Furthermore, DurIAN can be used to generate high quality facial expression which can be synchronized with generated speech with/without parallel speech and face data. To improve the efficiency of speech generation, we also propose a multi-band parallel generation strategy on top of the WaveRNN model. The proposed Multi-band WaveRNN effectively reduces the total computational complexity from 9.8 to 5.5 GFLOPS, and is able to generate audio that is 6 times faster than real time on a single CPU core. We show that DurIAN could generate highly natural speech that is on par with current state of the art end-to-end systems, while at the same time avoid word skipping/repeating errors in those systems. Finally, a simple yet effective approach for fine-grained control of expressiveness of speech and facial expression is introduced.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Fine-Grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis (2020, methodology): “To reconstruct waveforms from mel-spectrogram, we build a multi-band WaveRNN [22] trained by ground-truth mel-spectrogram for fair comparisons.”
Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, methodology): “A full-band audio was divided into four subbands, and by predicting the four subbands at the same time using the same network, the parameters of WaveRNN were significantly reduced.”
Parallel Tacotron: Non-Autoregressive and Controllable TTS (2020, methodology): “Among these duration-based models, some of them use the autoregressive decoder [23, 24], whereas others use a non-autoregressive decoder [15–22], which are more efficient during both training and inference.”
Didispeech: A Large Scale Mandarin Speech Corpus (2020, methodology): “The analysis filters and synthesis filters proposed in [30] are employed for multi-band processing.”
Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, methodology): “The DurIAN synthesis model used in this paper is inspired by [32], which is trained to predict the mel-spectrogram X from an input phoneme sequence Y , as shown in Fig.”
EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion (2021, methodology): “The backbone model is similar to that in DurIAN [11] and Pat-net [15], but our scheme differs in that we reﬁne the predicted duration and use the partial inference for speech generation in the editing scenario.”
Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, background): “Generally, with a well-trained neural acoustic model [2, 5, 6, 7] and a neural vocoder [8, 9, 10, 11], or alternatively using fully end-to-end models [12, 13, 14] which directly construct wave signals from text input, it is able to synthesize high-quality neutral speech.”
FeatherTTS: Robust and Efficient attention based Neural TTS (2020, methodology): “Even, many researches use hard alignment based on duration expansion instead of attention mechanism, such as FastSpeech [4], DurIAN [10].”
CUHK-EE voice cloning system for ICASSP 2021 M2VoC challenge (2021, methodology): “The DurIAN system [6] also uses the autoregressive model design and incorporates an explicit duration model.”
Controllable Emotion Transfer For End-to-End Speech Synthesis (2020, unknown): “Finally, we use a CBHG-based post-net to transform the mel-scale spectrogram into a linear spectrogram for reconstructing waveform by a multi-band WaveRNN [4].”
SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, methodology): “Waveforms segmentation has been used for generator acceleration in Multi-band WaveRNN (Yu et al. 2019) and Multi-band MelGAN, and here we employ it for multiband discrimination.”
ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, background): “One of the fastest GAN vocoders is multi-band (MB) MelGAN [24], which increases the speed of MelGAN [10] by changing the synthesis target from a full-band signal to lower-resolution sub-band signals [37].”
FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction (2020, methodology): “Tacotron 2 [5] and DurIAN [6], to achieve generating human-like speech.”
DurIAN: Duration Informed Attention Network for Speech Synthesis (2020, methodology): “Finally, the proposed DurIAN model is a general synthesis framework which we have successfully extended for other generation tasks such as singing [25, 26], multimodal synthesis [27], and finegrained style controlled speech synthesis [27].”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Since one-to-many regression problems like TTS can benefit from autoregressive decoding, it is also possible to combine the autoregressive method with duration predictor to further improve the stability of TTS models, such as the alignment methods used in DurIAN, Non-Attentive Tacotron [190], Device”
JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment (2020, methodology): “[10] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei, D. Su, and D. Yu, “DurIAN: Duration Informed Attention Network For Multimodal Synthesis,” ArXiv, vol. abs/1909.01700, 2019.”
DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech (2020, unknown): “In recent years, many excellent TTS systems have been proposed to simplify the traditional TTS pipeline and remove the dependencies of hand crafted intermediate representations [2–8].”
Improve GAN-based Neural Vocoder using Truncated Pointwise Relativistic Least Square GAN (2022, methodology): “To address the efficiency issue, many approaches are proposed to accelerate the inference speed of AR models, Yu et al.[35] modify the original WaveRNN[7] and divide the full-band audio signal into four subbands and predict the four subband parameters simultaneously, which leads to reduced predictio”