A Universal Music Translation Network

By Noam Mor, Lior Wolf, Adam Polyak, Yaniv Taigman. 2018

115 citations recorded by the source corpus.

We present a method for translating music across musical instruments, genres, and styles. This method is based on a multi-domain wavenet autoencoder, with a shared encoder and a disentangled latent space that is trained end-to-end on waveforms. Employing a diverse training dataset and large net capacity, the domain-independent encoder allows us to translate even from musical domains that were not seen during training. The method is unsupervised and does not rely on supervision in the form of matched samples between domains or musical transcriptions. We evaluate our method on NSynth, as well as on a dataset collected from professional musicians, and achieve convincing translations, even when translating from whistling, potentially enabling the creation of instrumental music by untrained humans.

Subfields: Computer Vision and Pattern Recognition, Music, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Music

In Signal Processing

Citations in the corpus

RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, background): “Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation (Oord et al., 2016; Vasquez & Lewis, 2019; Dhariwal et al., 2020) or timbre transfer between instruments (Mor et al., 2018).”
Neural Music Synthesis for Flexible Timbre Control (2018, background): “There are also a few notable applications of WaveNet on music, including NSynth [19], an autoencoder architecture which separately encodes monophonic pitch with learned timbral features, and the universal music translation network [20] which uses a denoising autoencoder architecture that can extract”
Disentangling speech from surroundings in a neural audio codec (2022, methodology): “This has been applied to disentanglement of image features [10], sensor anonymization [11], voice conversion [12, 13] and music translation [14].”
Unsupervised Singing Voice Conversion (2019, methodology): “The architecture of the encoder, decoder, and confusion network mostly reuse the successful WaveNet autoencoder architecture [2, 3].”
MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer (2018, background): “Mor et al. [26] develop a system based on a WaveNet au-toencoder [12] that can translate music across instruments, genres and styles, and even create music from whistling.”
TTS Skins: Speaker Conversion via ASR (2019, methodology): “The latter was shown in [14] to produce results of a better quality.”
Fast and Flexible Neural Audio Synthesis (2019, background): “This enables unique opportunities for timbre transfer and morphing [2,12], but the latent code is relatively high dimensional and unintuitive to manipulate due to entanglement of perceptual attributes.”
Latent Translation: Crossing Modalities by Bridging Generative Models (2019, background): “This has been demonstrated to great effect within a single modality, for example transferring between two different styles of image (Isola et al., 2017; Zhu et al., 2017; Li et al., 2018; Li, 2018), video (Wang et al., 2018), or music (Mor et al., 2018).”
Encoding Musical Style with Transformer Autoencoders (2019, background): “The Universal Music Translation network aims to translate music across various styles, but is not directly comparable to our approach as they work with raw audio waveforms (Mor et al., 2018).”
Deep Multimodal Representation Learning: A Survey (2019, background): “[83] proposed a model to translate music across”