RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion

By Dacheng Yin, Chuanxin Tang, Yanqing Liu, Xiaoqiang Wang, Zhiyuan Zhao, Yucheng Zhao, Zhiwei Xiong, Sheng Zhao, Chong Luo. 2022

14 citations recorded by the source corpus.

This paper proposes a new"decompose-and-edit"paradigm for the text-based speech insertion task that facilitates arbitrary-length speech insertion and even full sentence generation. In the proposed paradigm, global and local factors in speech are explicitly decomposed and separately manipulated to achieve high speaker similarity and continuous prosody. Specifically, we proposed to represent the global factors by multiple tokens, which are extracted by cross-attention operation and then injected back by link-attention operation. Due to the rich representation of global factors, we manage to achieve high speaker similarity in a zero-shot manner. In addition, we introduce a prosody smoothing task to make the local prosody factor context-aware and therefore achieve satisfactory prosody continuity. We further achieve high voice quality with an adversarial training stage. In the subjective test, our method achieves state-of-the-art performance in both naturalness and similarity. Audio samples can be found at https://ydcustc.github.io/retrieverTTS-demo/.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Tan
2. Chenpeng Du
3. Tao Qin
4. Yanqing Liu
5. Yiwei Guo
6. Kai Shen
7. Zeqian Ju
8. Sheng Zhao

Citations in the corpus

• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Lastly, for evaluating speech editing, we employ the same test set as utilized in (Yin et al. 2022).”
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (2023, methodology): “2) For the diffusion model, instead of directly attending to the hidden sequence from the prompt encoder that exposes too many details to the diffusion model and may harm the generation, we design two attention blocks: in the first attention block, we use m randomly initialized embeddings as the que”
• DiffVoice: Text-to-Speech with Latent Diffusion (2023, methodology): “We strictly followed the evaluation set up described in [17].”