A Neural Parametric Singing Synthesizer

By Merlijn Blaauw, Jordi Bonada. 2017

99 citations recorded by the source corpus.

We present a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Our model makes frame-wise predictions using mixture density outputs rather than categorical outputs in order to reduce the required parameter count. As we found overfitting to be an issue with the relatively small datasets used in our experiments, we propose a method to regularize the model and make the autoregressive generation process more robust to prediction errors. Using a simple multi-stream architecture, harmonic, aperiodic and voiced/unvoiced components can all be predicted in a coherent manner. We compare our method to existing parametric statistical and state-of-the-art concatenative methods using quantitative metrics and a listening test. While naive implementations of the autoregressive generation algorithm tend to be inefficient, using a smart algorithm we can greatly speed up the process and obtain a system that's competitive in both speed and quality.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, background): “…(Hiller Jr & Isaacson, 1957; Moorer, 1972; Hadjeres et al., 2017; Huang et al., 2017), digital vocoders generating a singer’s voice (Bonada & Serra, 2007; Saino et al., 2006; Blaauw & Bonada, 2017) and also synthesizers producing timbres for various musical instruments (Engel et al., 2017; 2019).”
DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (2021, methodology): “Nishimura et al. (2016); Blaauw and Bonada (2017); Kim et al. (2018); Nakamura et al. (2019); Gu et al. (2020) utilize neural networks to map the contextual features to acoustic features.”
MuseGAN: Symbolic-domain Music Generation and Accompaniment with Multi-track Sequential Generative Adversarial Networks (2017, background): “…et al. 2016; Bretan, Weinberg, and Heck 2016; Yang, Chou, and Yang 2017), requiring user input (Chu, Urtasun, and Fidler 2016), focusing on one specific genre (Hadjeres and Pachet 2016), or dealing with audio-domain, continuous-valued generation (van den Oord et al. 2016; Blaauw and Bonada 2017).”
Speaker-independent raw waveform model for glottal excitation (2018, methodology): “ density networks extend more easily to multivariate modeling: for example, a WaveNet-like architecture with Gaussian mixtures has been proposed for generating vocoder parameters in singing synthesis [19]. Res. block Res. block Res. block Res. block Res. block Res. block !"#' !"#&”
Unsupervised Singing Voice Conversion (2019, methodology): “Blaaauw and Bonda have demonstrated very convincing singing synthesis using a WaveNet Decoder [16].”
WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN (2019, methodology): “Modelling the timbre independently of the pitch has been shown to be an effective methodology [2].”
When Machine Learning Meets Privacy (2020, methodology): “A modified version of the proposed model is used for singing synthesis in [11].”
Unsupervised Cross-Domain Singing Voice Conversion (2020, methodology): “Recent works demonstrated the need of both linguistic and musical features [19, 20, 30] in the context of singing generation.”
Improving FFTNet Vocoder with Noise Shaping and Subband Approaches (2018, methodology): “The same strategy is used in several other methods: 1) pixelCNN [54, 55] for conditional color image generation, where a blue pixel is predicted from the previous red, green, and blue pixels; 2) the neural parametric singing synthesizer [56,57], where Overlapped SSB synthesis ﬁlterbank”