What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS

By Brooke Stephenson, Laurent Besacier, Laurent Girin, Thomas Hueber. 2020

16 citations recorded by the source corpus.

In incremental text to speech synthesis (iTTS), the synthesizer produces an audio output before it has access to the entire input sentence. In this paper, we study the behavior of a neural sequence-to-sequence TTS system when used in an incremental mode, i.e. when generating speech output for token n, the system has access to n + k tokens from the text sequence. We first analyze the impact of this incremental policy on the evolution of the encoder representations of token n for different values of k (the lookahead parameter). The results show that, on average, tokens travel 88% of the way to their full context representation with a one-word lookahead and 94% after 2 words. We then investigate which text features are the most influential on the evolution towards the final representation using a random forest analysis. The results show that the most salient factors are related to token length. We finally evaluate the effects of looka-head k at the decoder level, using a MUSHRA listening test. This test shows results that contrast with the above high figures: speech synthesis quality obtained with 2 word-lookahead is significantly lower than the one obtained with the full sentence.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Xinyu Yang
6. Zhaoxi Mu
7. Yizhuo Dong
8. Sheng Zhao

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Streaming TTS [74, 217, 316, 399, 317, 231] synthesizes speech once some input tokens are comming, without waiting for the whole input sentence, which can also speed up inference.”
• Review of end-to-end speech synthesis technology based on deep learning (2021, background): “To solve this problem, some researchers have proposed streaming incremental TTS systems [50, 133, 198, 235], which can output speech in real time while inputting text, because they only need to see a few characters or words to synthesize speech.”
• Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input (2021, result): “Importantly, we force GPT-2 predictions and random predictions to have comparable lengths in term of characters/phonemes because (1) GPT-2 tends to predict shorter words because they are more frequent, (2) in our previous study [8], we found that longer future words have more influence on the curren”