Bi-Level Speaker Supervision for One-Shot Speech Synthesis

By Jianhua Tao, Ruibo Fu, Jiangyan Yi, Zhengqi Wen, Chunyu Qiang. 2020

10 citations recorded by the source corpus.

The gap between speaker characteristics of reference speech and synthesized speech remains a challenging problem in one-shot speech synthesis. In this paper, we propose a bi-level speaker supervision framework to close the speaker characteristics gap via supervising the synthesized speech at speaker feature level and speaker identity level. The speaker feature extraction and speaker identity reconstruction are integrated in an end-to-end speech synthesis network, with the one on speaker feature level for closing speaker characteristics and the other on speaker identity level for preserving identity information. This framework guarantees that the synthesized speech has similar speaker characteristics to original speech, and it also ensures the distinguishability between different speakers. Additionally, to solve the influence of speech content on speaker feature extraction task, we propose a text-independent reference encoder (ti-reference encoder) module to extract speaker feature. Experiments on LibriTTS dataset show that our model is able to generate the speech similar to target speaker. Furthermore, we demonstrate that this model can learn meaningful speaker representations by bi-level speaker supervision and ti-reference encoder module.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Da-Rong Liu
2. Yi-Chen Chen
3. Sung-Feng Huang
4. Hung-yi Lee
5. Chyi-Jiunn Lin
6. James R. Glass
7. Xu Tan
8. Tao Qin

In Signal Processing

1. Yi-Chen Chen
2. Sung-Feng Huang
3. Da-Rong Liu
4. Hung-yi Lee
5. Chyi-Jiunn Lin
6. Yu-An Chung
7. Lin-Shan Lee
8. James R. Glass

Citations in the corpus

• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, neutral): “Some previous works [19, 20] apply speaker classification loss on generated mel-spectrograms as a supervision to ensure the synthesized speech to be more similar to reference speech, which, however, does not bring much improvement in speaker similarity [16].”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, background): “There are two general approaches to deal with such task [8], speaker adaptation [8], [9], [10], [11], [12] and speaker encoding [6], [13], [7], [14], [15], [16], [17].”