Deep Learning Techniques for Music Generation - A Survey

By Jean-Pierre Briot, Gaëtan Hadjeres, François Pachet. 2017

330 citations recorded by the source corpus.

This paper is a survey and an analysis of different ways of using deep learning (deep artificial neural networks) to generate musical content. We propose a methodology based on five dimensions for our analysis: Objective - What musical content is to be generated? Examples are: melody, polyphony, accompaniment or counterpoint. - For what destination and for what use? To be performed by a human(s) (in the case of a musical score), or by a machine (in the case of an audio file). Representation - What are the concepts to be manipulated? Examples are: waveform, spectrogram, note, chord, meter and beat. - What format is to be used? Examples are: MIDI, piano roll or text. - How will the representation be encoded? Examples are: scalar, one-hot or many-hot. Architecture - What type(s) of deep neural network is (are) to be used? Examples are: feedforward network, recurrent network, autoencoder or generative adversarial networks. Challenge - What are the limitations and open challenges? Examples are: variability, interactivity and creativity. Strategy - How do we model and control the process of generation? Examples are: single-step feedforward, iterative feedforward, sampling or input manipulation. For each dimension, we conduct a comparative analysis of various models and techniques and we propose some tentative multidimensional typology. This typology is bottom-up, based on the analysis of many existing deep-learning based systems for music generation selected from the relevant literature. These systems are described and are used to exemplify the various choices of objective, representation, architecture, challenge and strategy. The last section includes some discussion and some prospects.

Subfields: Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Speech & Audio

Citations in the corpus

The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “More recently, deep learning and modern generative modelling techniques have been applied to this task [5, 7].”
Differentially Private Learning with Adaptive Clipping (2019, neutral): “Deep learning has become ubiquitous, with applications as diverse as image processing, natural language translation, and music generation (Szegedy et al., 2015; He et al., 2015; Vinyals et al., 2015; Briot et al., 2017).”
Learning a Latent Space of Multitrack Measures (2018, background): “The use of neural networks in symbolic music generation has seen a resurgence in interest, as surveyed by Briot et al. [4].”
MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment (2017, background): “As reviewed by (Briot, Hadjeres, and Pachet 2017), a surging number of models have been proposed lately for symbolic music generation.”
Deep Reinforcement Learning: An Overview (2017, background): “We do not discuss some interesting applications, like music generation (Briot et al., 2017; Jaques et al., 2017), and retrosynthesis (Segler et al.”
Human-AI Co-Creation in Songwriting (2020, background): “More recently, a proliferation of modern deep learning techniques [8,9] has enabled models capable of generating full scores [40], or producing music that is coherent to both local and distant regions of music [39, 55].”
Artificial intelligence in the creative industries: a review (2020, background): “This makes them amenable to tasks, such as speech recognition [325], handwriting recognition [326], and music generation [228].”
Deep Learning for Audio Signal Processing (2019, background): “[145] provide a more in-depth treatment of music generation with deep learning.”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “tion or a sketch to the assistant for creating an image [180], [237]; and 4) lyrics and score to the assistant for composing a piece of music [238].”
MuseGAN: Symbolic-domain Music Generation and Accompaniment with Multi-track Sequential Generative Adversarial Networks (2017, background): “Music Generation As reviewed by (Briot, Hadjeres, and Pachet 2017), a surging number of neural networks have been proposed lately for music generation.”
Generating Interactive Worlds with Text (2019, background): “There has been work in ML for music generation, see e.g. Magenta2 or Briot, Hadjeres, and Pachet (2017) for a survey.”
Differentially Private Learning with Adaptive Clipping (2019, background): “Deep learning has found a variety of applications, including image classification, language translation, and music generation [25, 12, 26, 4].”
This time with feeling: learning expressive musical performance (2018, background): “Briot et al [6] provide a survey of generative music models that involve machine learning.”
Human-in-the-loop Artificial Intelligence (2017, background): “Intelligent machines may produce music jingles for commercials [5], write novels, produce news articles and so on.”
A Gentle Introduction to Deep Learning in Medical Image Processing (2018, background): “ts in speech recognition [2] and image recognition [3], it is able to train articial agents that beat human players in Go [4] and ATARI games [5], and it creates artistic new images [6, 7] and music [8]. Many of these tasks were considered to be impossible to be solved by computers before the advent”