Many-to-Many Voice Transformer Network

By Hirokazu Kameoka, Wen-Chin Huang, Kou Tanaka, Takuhiro Kaneko, Nobukatsu Hojo, Tomoki Toda. 2020

33 citations recorded by the source corpus.

This paper proposes a voice conversion (VC) method based on a sequence-to-sequence (S2S) learning framework, which enables simultaneous conversion of the voice characteristics, pitch contour, and duration of input speech. We previously proposed an S2S-based VC method using a transformer network architecture called the voice transformer network (VTN). The original VTN was designed to learn only a mapping of speech feature sequences from one speaker to another. Here, the main idea we propose is an extension of the original VTN that can simultaneously learn mappings among multiple speakers. This extension, called the many-to-many VTN, enables us to fully use available training data collected from multiple speakers by capturing common latent features that can be shared across different speakers. It also allows us to introduce a training loss called the identity mapping loss to ensure that the input feature sequence will remain unchanged when the source and target speaker indices are the same. Using this particular loss for model training has been found to be extremely effective in improving the performance of the model at test time. We conducted speaker identity conversion experiments and found that our model obtained higher sound quality and speaker similarity than baseline methods. We also found that our model, with a slight modification to its architecture, can handle any-to-many conversion tasks reasonably well.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Junichi Yamagishi
4. Yu Wu
5. Hirokazu Kameoka
6. Furu Wei
7. Long Zhou
8. Tomoki Toda

In Signal Processing

1. Jinyu Li
2. Hirokazu Kameoka
3. Shujie Liu
4. Tomoki Toda
5. Junichi Yamagishi
6. Tomoki Hayashi
7. Yi-Chiao Wu
8. Yu Wu

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “The WER is evaluated by using the public ASR model HuBERT L ARGE 8 , where the WER of the test set with this ASR model is comparable to that of VTN (Huang et al., 2021).”
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, methodology): “In the future, we plan to extend our pretraining technique to more flexible training conditions, such as many-to-many [72] or nonparallel training [36].”
• Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, unknown): “and the non-parallel seq2seq methods [13], [14].”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “TransformerTTS [10] first adopted a transformer network for TTS, and VTN [11] also uses a transformer network for VC.”