Neural Music Synthesis for Flexible Timbre Control

By Jong Wook Kim, Rachel M. Bittner, Aparna Kumar, Juan Pablo Bello. 2018

44 citations recorded by the source corpus.

The recent success of raw audio waveform synthesis models like WaveNet motivates a new approach for music synthesis, in which the entire process — creating audio samples from a score and instrument information — is modeled using generative neural networks. This paper describes a neural music synthesis model with flexible timbre controls, which consists of a recurrent neural network conditioned on a learned instrument embedding followed by a WaveNet vocoder. The learned embedding space successfully captures the diverse variations in timbres within a large dataset and enables timbre control and morphing by interpolating between instruments in the embedding space. The synthesis quality is evaluated both numerically and perceptually, and an interactive web demo is presented.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Ilya Sutskever

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, methodology): “…synthesizing music conditioned on symbolic music information, such as NSynth (Engel et al., 2017) which uses WaveNet-style autoen-coder, Mel2Mel (Kim et al., 2019) and Wave2Midi2Wave (Hawthorne et al., 2019) which synthesize music using WaveNet conditioned on a piano roll representation, and…”