HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis

By Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, Tie-Yan Liu. 2020

114 citations recorded by the source corpus.

High-fidelity singing voices usually require higher sampling rate (e.g., 48kHz) to convey expression and emotion. However, higher sampling rate causes the wider frequency band and longer waveform sequences and throws challenges for singing voice synthesis (SVS) in both frequency and time domains. Conventional SVS systems that adopt small sampling rate cannot well address the above challenges. In this paper, we develop HiFiSinger, an SVS system towards high-fidelity singing voice. HiFiSinger consists of a FastSpeech based acoustic model and a Parallel WaveGAN based vocoder to ensure fast training and inference and also high voice quality. To tackle the difficulty of singing modeling caused by high sampling rate (wider frequency band and longer waveform), we introduce multi-scale adversarial training in both the acoustic model and vocoder to improve singing modeling. Specifically, 1) To handle the larger range of frequencies caused by higher sampling rate, we propose a novel sub-frequency GAN (SF-GAN) on mel-spectrogram generation, which splits the full 80-dimensional mel-frequency into multiple sub-bands and models each sub-band with a separate discriminator. 2) To model longer waveform sequences caused by higher sampling rate, we propose a multi-length GAN (ML-GAN) for waveform generation to model different lengths of waveform sequences with separate discriminators. 3) We also introduce several additional designs and findings in HiFiSinger that are crucial for high-fidelity voices, such as adding F0 (pitch) and V/UV (voiced/unvoiced flag) as acoustic features, choosing an appropriate window/hop size for mel-spectrogram, and increasing the receptive field in vocoder for long vowel modeling. Experiment results show that HiFiSinger synthesizes high-fidelity singing voices with much higher quality: 0.32/0.44 MOS gain over 48kHz/24kHz baseline and 0.83 MOS gain over previous SVS systems.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Yi Ren
6. Sheng Zhao
7. Jinglin Liu
8. Frank K. Soong

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Yi Ren
6. Jinglin Liu
7. Rongjie Huang
8. Sheng Zhao

Citations in the corpus

• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “We adopt the multi-length discriminator [10, 63] based on random windows of different lengths as the discriminator.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Due to page limitations, we only reviewed core algorithms of TTS; readers can refer to other papers for TTS related problems and applications, such as voice conversion [302], singing voice synthesis [112, 211, 35], talking face synthesis [36], etc.”
• DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (2021, methodology): “As shown in Figure 5, we compare the ground truth, the generated mel-spectrograms from Diffsinger, GAN-singer and FFT-Singer with the same music score.”
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “To ensure high fidelity and quality of the synthesized speech, we directly generate waveform in 48 kHz sampling rate, since it can have larger range of frequency to convey expression and prosody [16].”
• Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, methodology): “This adversarial training method is similar to the sub-frequency adversarial loss in HifiSinger [23] but with an extra multi-length adversarial loss on the spectrogram.”
• A Survey on Recent Deep Learning-driven Singing Voice Synthesis Systems (2021, methodology): “This mechanism utilizes a simple auxiliary decoder seen in Transformer-based acoustic models [2], [6] to generate a rough Mel-spectrogram from the musical score.”
• DiffSinger: Diffusion Acoustic Model for Singing Voice Synthesis (2021, methodology): “In the inference stage, for all the experiments, we uniformly use a pretrained Parallel WaveGAN (PWG) (Yamamoto et al., 2020) 11 as vocoder to transform the generated mel-spectrograms into waveforms (audio samples).”
• NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, background): “Another issue is that generating high-fidelity singing voice from mel spectrogram is well known to be difficult owing to a wide pitch range, long continuous pronunciation, and high sampling rate (Chen et al., 2020a; Perrotin et al., 2021; Morrison et al., 2022).”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, methodology): “Most previous works focus on optimizing the acoustic model, but usually use speech vocoders for SVS (Gu et al. 2020; Chen et al. 2020).”
• Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus (2021, unknown): “Singing voice synthesis (SVS) systems [2, 14, 22] take music score and lyric information as input to generate singing voices, and these systems have been widely deployed in music softwares, music boxes, and so on.”
• M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus (2022, background): “…learning and the continuous breakthroughs in TTS techniques [47; 23; 52; 37; 35; 36; 18], singing voice synthesis [45; 21; 14; 5; 38; 2; 27; 40; 9; 13; 10; 51; 17; 8; 43; 1; 22; 48; 54; 41; 19; 56; 53; 25] and its related tasks (e.g., SVC [33; 42; 24; 46; 55]) have received extensive attention…”