SING: Symbol-to-Instrument Neural Generator

By Alexandre D'efossez, Neil Zeghidour, Nicolas Usunier, L. Eon Bottou, Francis R. Bach. 2018

63 citations recorded by the source corpus.

Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as WaveNet and SampleRNN suffer from prohibitive training and inference times because they are based on autoregressive models that generate audio samples one at a time at a rate of 16kHz. In this work, we study the more computationally efficient alternative of generating the waveform frame-by-frame with large strides. We present a lightweight neural audio synthesizer for the original task of generating musical notes given desired instrument, pitch and velocity. Our model is trained end-to-end to generate notes from nearly 1000 instruments with a single decoder, thanks to a new loss function that minimizes the distances between the log spectrograms of the generated and target waveforms. On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

DDSP: Differentiable Digital Signal Processing (2020, methodology): “It also makes them incompatible with perceptual losses such as spectral features (Defossez et al., 2018), pretrained models (Dosovitskiy & Brox, 2016), and discriminators (Engel et al., 2019).”
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, methodology): “This precludes real-time application which are pervasive in musical creation, while parallel models (Défossez et al., 2018) can only allow fast generation at the cost of a lower sound quality.”
Music Source Separation in the Waveform Domain (2019, methodology): “015; Stoller et al., 2018; Jansson et al., 2017), and builds on transposed convolutions with large number of channels and large strides (4) inspired by the approach to the synthesis of music notes of Défossez et al. (2018). The U-Net skip connections and the gating performed by GLUs imply that this ”
GANSynth: Adversarial Neural Audio Synthesis (2019, methodology): “Follow up work has explored diverse approaches including frame-based regression models(Defossez et al., 2018), inverse scattering networks(Andreux & Mallat, 2018), VAEs with perceptual priors(Esling et al.”