Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework

By Mingbo Ma, Baigong Zheng, Kaibo Liu, Renjie Zheng, Hairong Liu, Kainan Peng, Kenneth Ward Church, Liang Huang. 2019

36 citations recorded by the source corpus.

Text-to-speech synthesis (TTS) has witnessed rapid progress in recent years, where neural methods became capable of producing audios with high naturalness. However, these efforts still suffer from two types of latencies: (a) the computational latency (synthesizing time), which grows linearly with the sentence length, and (b) the input latency in scenarios where the input text is incrementally available (such as in simultaneous translation, dialog generation, and assistive technologies). To reduce these latencies, we propose a neural incremental TTS approach using the prefix-to-prefix framework from simultaneous translation. We synthesize speech in an online fashion, playing a segment of audio while generating the next, resulting in an O(1) rather than O(n) latency. Experiments on English and Chinese TTS show that our approach achieves similar speech naturalness compared to full sentence TTS, but only with a constant (1-2 words) latency.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Juan Pino
5. Wei-Ning Hsu
6. Frank K. Soong
7. Ann Lee
8. Sheng Zhao

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Streaming TTS [74, 217, 316, 399, 317, 231] synthesizes speech once some input tokens are comming, without waiting for the whole input sentence, which can also speed up inference.”
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search (2020, background): “As the synthesizing time of the models grows linearly with the output length, undesirable delay caused by generating long speech can be propagated to the multiple pipelines of TTS systems without designing sophisticated frameworks (Ma et al., 2019).”
• Review of end-to-end speech synthesis technology based on deep learning (2021, background): “[133] borrowed the idea of prefix-to-prefix framework of simultaneous machine translation [132].”
• Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning (2020, methodology): “[17] proposes an approach for incremental neural TTS.”
• Direct simultaneous speech to speech translation (2021, methodology): “However, such estimation is non-trivial, since inference time alignment is not differentiable during training.”
• What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS (2020, methodology): “We define an iTTS system with the following simple policy (similar to [16]): the encoder starts by reading k input tokens (k is the lookahead parameter) and then it alternates between generating speech output and reading the next token until the complete input token sequence is consumed.”