Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning

By Devang S. Ram Mohan, Raphael Lenain, Lorenzo Foglianti, Tian Huey Teh, Marlene Staib, Alexandra Torresquintero, Jiameng Gao. 2020

11 citations recorded by the source corpus.

Modern approaches to text to speech require the entire input character sequence to be processed before any audio is synthesised. This latency limits the suitability of such models for time-sensitive tasks like simultaneous interpretation. Interleaving the action of reading a character with that of synthesising audio reduces this latency. However, the order of this sequence of interleaved actions varies across sentences, which raises the question of how the actions should be chosen. We propose a reinforcement learning based framework to train an agent to make this decision. We compare our performance against that of deterministic, rule-based systems. Our results demonstrate that our agent successfully balances the trade-off between the latency of audio generation and the quality of synthesised audio. More broadly, we show that neural sequence-to-sequence models can be adapted to run in an incremental manner.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Yi Ren
7. Zhou Zhao
8. Lei He

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “Streaming TTS [74, 217, 316, 399, 317, 231] synthesizes speech once some input tokens are comming, without waiting for the whole input sentence, which can also speed up inference.”
• Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input (2021, unknown): “In this new paradigm, the tradeoff between speech quality and synthesis latency has been examined by testing the effects of different degrees of lookahead [8, 9], reinforcement learning has been used to automatically learn a wait/synthesize policy [10], the effects of synthesis unit have been studie”