High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency

By Nikolaos Ellinas, Georgios Vamvoukakis, Konstantinos Markopoulos, Aimilios Chalamandaris, Georgia Maniati, Panos Kakoulidis, Spyros N. Raptis, June Sig Sung, Hyoungmin Park, Pirros Tsiakoulis. 2020

41 citations recorded by the source corpus.

This paper presents an end-to-end text-to-speech system with low latency on a CPU, suitable for real-time applications. The system is composed of an autoregressive attention-based sequence-to-sequence acoustic model and the LPCNet vocoder for waveform generation. An acoustic model architecture that adopts modules from both the Tacotron 1 and 2 models is proposed, while stability is ensured by using a recently proposed purely location-based attention mechanism, suitable for arbitrary sentence length generation. During inference, the decoder is unrolled and acoustic feature generation is performed in a streaming manner, allowing for a nearly constant latency which is independent from the sentence length. Experimental results show that the acoustic model can produce feature sequences with minimal latency about 31 times faster than real-time on a computer CPU and 6.5 times on a mobile CPU, enabling it to meet the conditions required for real-time applications on both devices. The full end-to-end system can generate almost natural quality speech, which is verified by listening tests.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Sheng Zhao

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Frank K. Soong
5. Zhaoxi Mu
6. Xinyu Yang
7. Yizhuo Dong
8. Seongkyu Mun

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Speedup with Domain Knowledge Domain knowledge from speech can be leveraged to speed up inference, such as linear prediction [357], multiband modeling [411, 400], subscale prediction [147], multi-frame prediction [420, 376, 367, 123], streaming synthesis [74], etc.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[50] proposed a streaming inference method, which can input the generated acoustic frames into the vocoder before the inference process of the acoustic model is completed.”
• Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation (2022, background): “Input phonemes are converted to acoustic features for the LPCNet vocoder [26], which has been shown to produce high quality results in our previous work [27].”
• An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Space (2022, methodology): “We choose a Tacotron [13, 14] variant [15] as the representative model for the former, and Glow-TTS [16] for the latter.”