Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus

By Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, Zhou Zhao. 2021

135 citations recorded by the source corpus.

High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost. Existing open corpora could not meet requirements for high-fidelity singing voice synthesis because of the scale and quality weaknesses. Previous vocoders have difficulty in multi-singer modeling, and a distinct degradation emerges when conducting unseen singer singing voice generation. To accelerate singing voice researches in the community, we release a large-scale, multi-singer Chinese singing voice dataset OpenSinger. To tackle the difficulty in unseen singer modeling, we propose Multi-Singer, a fast multi-singer vocoder with generative adversarial networks. Specifically, 1) Multi-Singer uses a multi-band generator to speed up both training and inference procedure. 2) to capture and rebuild singer identity from the acoustic feature (i.e., mel-spectrogram), Multi-Singer adopts a singer conditional discriminator and conditional adversarial training objective. 3) to supervise the reconstruction of singer identity in the spectrum envelopes in frequency domain, we propose an auxiliary singer perceptual loss. The joint training approach effectively works in GANs for multi-singer voices modeling. Experimental results verify the effectiveness of OpenSinger and show that Multi-Singer improves unseen singer singing voices modeling in both speed and quality over previous methods. The further experiment proves that combined with FastSpeech 2 as the acoustic model, Multi-Singer achieves strong robustness in the multi-singer singing voice synthesis pipeline.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Rongjie Huang
3. Zhou Zhao
4. Yi Ren
5. Jinyu Li
6. Jinglin Liu
7. Shinji Watanabe
8. Chao Weng

In Signal Processing

1. Rongjie Huang
2. Dong Yu
3. Zhou Zhao
4. Jinyu Li
5. Yi Ren
6. Jinglin Liu
7. Shinji Watanabe
8. Chenye Cui

Citations in the corpus

• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “Compared to recent SOTA models, such as Tango and Make-an-Audio 2, UniAudio also got comparable performance.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “, Tacotron [50], FastSpeech [36]), which is then transformed to waveform using a separately trained vocoder [9, 19], or directly generate waveform from text (e.”
• Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, methodology): “…Works 2.1 Speech/Singing Synthesis TTS or SVS models typically convert input text into mel-spectrogram (e.g., Tacotron [40], Fast-Speech [33]), which is then transformed to waveform using a separately trained vocoder [20, 12], or directly generate waveform from text (e.g., EATS [10] and VITS [19]).”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “A growing number of applications [51, 11, 50, 52], such as voice assistant services and long-form reading, have been actively developed and deployed to real-world speech platforms.”
• M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus (2022, background): “…singing recordings are originally scraped with accompaniment and difficult to extract the clean vocals, which finally lead to great decline in audio quality for SVS. OpenSinger [17] is a wonderful SVS corpus with a large amount of high quality songs, and rich choices in singers and singing styles.”
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, unknown): “In this part, we decompose speech variations (Cui et al., 2022; Huang et al., 2021; Yang et al., 2022a) into linguistic content and acoustic condition (e.g., speaker identity, rhythm, pitch, and energy) and provide a brief primer on each of these components.”
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “…applications in text processing and generation, replicating this success for audio modality (speech (Ren et al., 2020; Huang et al., 2022a), music (Huang et al., 2021; Liu et al., 2022a), sound (Yang et al., 2022; Huang et al., 2023a), and talking head (Wu et al., 2021; Ye et al., 2023)) is…”
• Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, background): “…[29, 30] or other languages [31] have been released and also lots of research has been conducted towards Mandarin singing voice synthesis [24, 32, 33], no accessible public Mandarin SVS corpus with high quality exists, thereby limiting the development of Mandarin SVS. Note that although Chinese…”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, background): “[7] does not implicitly focus upon the characteristics of the singing voices but aims to alleviate the model generalization in the multi-singer scenario.”