Pretraining Techniques for Sequence-to-Sequence Voice Conversion

By Wen-Chin Huang, Tomoki Hayashi, Yi-Chiao Wu, Hirokazu Kameoka, Tomoki Toda. 2020

49 citations recorded by the source corpus.

Sequence-to-sequence (seq2seq) voice conversion (VC) models are attractive owing to their ability to convert prosody. Nonetheless, without sufficient data, seq2seq VC models can suffer from unstable training and mispronunciation problems in the converted speech, thus far from practical. To tackle these shortcomings, we propose to transfer knowledge from other speech processing tasks where large-scale corpora are easily available, typically text-to-speech (TTS) and automatic speech recognition (ASR). We argue that VC models initialized with such pretrained ASR or TTS model parameters can generate effective hidden representations for high-fidelity, highly intelligible converted speech. In this work, we examine our proposed method in a parallel, one-to-one setting. We employed recurrent neural network (RNN)-based and Transformer based models, and through systematical experiments, we demonstrate the effectiveness of the pretraining scheme and the superiority of Transformer based models over RNN-based models in terms of intelligibility, naturalness, and similarity.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Yu Wu
4. Furu Wei
5. Long Zhou
6. Tom Ko
7. Dong Yu
8. Yao Qian

In Signal Processing

1. Jinyu Li
2. Yu Wu
3. Shujie Liu
4. Furu Wei
5. Long Zhou
6. Helen Mei-Ling Meng
7. Xunying Liu
8. Xixin Wu

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “For each speaker, the first 932, the last 100, and the rest 100 sentences of the 1,132 sentences are used for training, test, and validation as (Huang et al., 2021), respectively.”
• Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, methodology): “A recent study, which is related to the BNE-Seq2seqMoL proposed in this paper, uses a pre-trained ASR encoder and a pre-trained TTS decoder to initialize parameters of the ultimate encoder-decoder based VC model [33].”