A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

By Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, Douglas Eck. 2018

554 citations recorded by the source corpus.

The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem, which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model. An implementation of our "MusicVAE" is available online at this http URL.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, background): “, 2018) which use generative adversarial networks, MusicVAE (Roberts et al., 2018) and HRNN (Wu et al.”
Symbolic Music Generation with Diffusion Models (2021, methodology): “Sequence VAEs such as MusicVAE are difficult to train on long sequences (Roberts et al., 2019), which we overcome by pairing the short 2-bar MusicVAE model with a diffusion model capable of modeling dependencies between 32 latents, thus modeling 64 bars in total.”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Beyond raw audio generation, several generative models that capture large-scale structure have been proposed for images [2, 3, 12, 27], dialogue [40], 3D shapes [30] and symbolic music [37].”
Machine learning in protein engineering (2018, background): “Tantalizingly, generative models in other ﬁelds have been trained to generate new faces (Radford et al., 2015), sketches (Ha & Eck, 2017), and even music (Roberts et al., 2018).”
LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training (2019, unknown): “Recent work [21–23] attempts multi-instrumental music generation from scratch, but these methods are limited to generating fixed lengths, unlike our method which can generate arbitrarily-long sequences.”
Pop Music Transformer: Generating Music with Rhythm and Harmony (2020, methodology): “To eliminate the imprecision of transcription result, we further quantize the onset and offset times to the 16th note, a commonly-adopted note interval in recent work on automatic music composition (e.g., [Roberts et al. , 2018]).”
Latent Translation: Crossing Modalities by Bridging Generative Models (2019, background): “…2017; Bikowski et al., 2018) including conditional generation (Mirza & Osindero, 2014), generation of one domain conditioned on another (Dai et al., 2017; Reed et al., 2016), generation of high-quality images (Karras et al., 2018), audio (Engel et al., 2019; 2017), and music (Roberts et al., 2018).”
Consistency Regularization for Variational Auto-Encoders (2021, background): “, 2019), music generation (Roberts et al., 2018), topic modeling (Miao et al.”
Dynamical Variational Autoencoders: A Comprehensive Review (2020, background): “The same principle has been applied for music scores modeling (Roberts et al., 2018).”
A Hierarchical Recurrent Neural Network for Symbolic Melody Generation (2017, methodology): “rocessed in parallel (Colombo et al. 2016). This approach is further extended in (Colombo, Seeholzer, and Gerstner 2017). A hierarchical VAE is employed to learn the distribution of melody pieces in (Roberts et al. 2018), the decoder of which is similar to our model. The major difference is that the”
Deep Learning Techniques for Music Generation - A Survey (2017, methodology): “Reproduced from [149] with the permission of the authors”
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models (2020, background): “For example, there are emerging approaches aimed at learning a continuous latent space so that users can interpolate between music [38], or explore a space of musical alternatives [10].”
Human-AI Co-Creation in Songwriting (2020, methodology): “For example, in the symbolic domain, the band Yacht used MusicVAE [60], a variational autoencoder type of neural network, to ﬁnd melodies hidden in between songs by interpolating between the musical lines in their back cata-log, commenting that “it took risks maybe we aren’t willing to” [47].”
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “For example, if we consider a 16-th note time grid as [23, 34], Q = 16.”
Learning a Latent Space of Multitrack Measures (2018, methodology): “In this section, we give a brief overview of our model’s architecture; for speciﬁc details refer to [33] and our public source code 1 .”
Encoding Musical Style with Transformer Autoencoders (2019, background): “The MusicVAE (Roberts et al., 2018) is a sequential VAE with a hierarchical recurrent decoder, which learns an interpretable latent code for musical sequences that can be used during generation time.”
Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning (2019, background): “Compared with traditional deterministic auto-encoder, the VAE offers two main advantages in our approach: (1) Deterministic auto-encoders often have “holes” in their latent space, where the latent representations may not able to generate anything realistic (Roberts et al. 2018).”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, background): “Advances in amortized variational inference [1, 2, 3, 4] have enabled novel learning methods [4, 5, 6] and extended generative learning into complex domains such as molecule design [7, 8], music [9] and program [10] generation.”
PIANOTREE VAE: Structured Representation Learning for Polyphonic Music (2020, methodology): “Specifically, • The piano-roll recurrent VAE ( pr-rnn ) model is similar to a 2-bar MusicVAE proposed in [4].”
Modeling Self-Repetition in Music Generation using Generative Adversarial Networks (2019, background): “Some prior works focus on modeling long term structures in music generation (Eck & Schmidhuber, 2002; Huang et al., 2018; Roberts et al., 2018).”