Unsupervised Cross-Domain Singing Voice Conversion

By Adam Polyak, Lior Wolf, Yossi Adi, Yaniv Taigman. 2020

48 citations recorded by the source corpus.

We present a wav-to-wav generative model for the task of singing voice conversion from any identity. Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator. The proposed generative architecture is invariant to the speaker's identity and can be trained to generate target singers from unlabeled training data, using either speech or singing sources. The model is optimized in an end-to-end fashion without any manual supervision, such as lyrics, musical notes or parallel samples. The proposed approach is fully-convolutional and can generate audio in real-time. Experiments show that our method significantly outperforms the baseline methods while generating convincingly better audio samples than alternative attempts.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “These representations are being used together with the identity and prosodic information for voice conversion [24, 25, 26].”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “Another line of works consists of modifying the parameters of the speech signal (Polyak et al., 2020b,a; Sisman et al., 2020; Robinson et al., 2019; Zhou et al., 2021b), i.e, Speech-to-Speech.”
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, unknown): “Speech voice conversion [2, 35, 44, 43] and singing voice conversion [7, 28] aim to modify the timbre without changing the speech or singing content.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “Similarly to (Polyak et al., 2020), we report Gross Pitch Error (GPE) (Nakatani et al.”
StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion (2021, methodology): “Similar to [16], we use the output from the intermediate layer before the LSTM layers as the linguistic feature, denoted by hasr(·).”
M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus (2022, unknown): “…in TTS techniques [47; 23; 52; 37; 35; 36; 18], singing voice synthesis [45; 21; 14; 5; 38; 2; 27; 40; 9; 13; 10; 51; 17; 8; 43; 1; 22; 48; 54; 41; 19; 56; 53; 25] and its related tasks (e.g., SVC [33; 42; 24; 46; 55]) have received extensive attention in both industry and academic communities.”