Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset

By Curtis Hawthorne, A. Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, Douglas Eck. 2018

596 citations recorded by the source corpus.

Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

In Signal Processing

Citations in the corpus

Music Transformer: Generating Music with Long-Term Structure (2018, background): “MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) (Hawthorne et al., 2019) is a dataset consisting of 172 hours of virtuosic piano performances captured in both MIDI”
Jukebox: A Generative Model for Music (2020, methodology): “…music information, such as NSynth (Engel et al., 2017) which uses WaveNet-style autoen-coder, Mel2Mel (Kim et al., 2019) and Wave2Midi2Wave (Hawthorne et al., 2019) which synthesize music using WaveNet conditioned on a piano roll representation, and GanSynth (Engel et al., 2019) which uses…”
Symbolic Music Generation with Diffusion Models (2021, methodology): “Wave2Midi2Wave [31] uses a transformer to generate MIDI-like symbolic data and then a WaveNet [32] to synthesize that symbolic data into audio.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “At inference, we extract a 4-second prompt from the Maestro dataset [62].”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, methodology): “We utilize the MAESTRO dataset [19], which consists of over 172 hours of solo piano performances.”
MusicLM: Generating Music From Text (2023, background): “Symbolic representations of music (e.g., MIDI) can also be used to drive the generative process as a form of strong conditioning, as demonstrated by Huang et al. (2019); Hawthorne et al. (2019); Engel et al. (2020).”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “To showcase the long-term coherence of these models, we introduce another symbolic task involving a larger, private dataset of piano performances (Simon et al., 2019).”
Encoding Musical Style with Transformer Autoencoders (2019, methodology): “The MAESTRO (Hawthorne et al., 2019) dataset consists of over 1,100 classical piano performances, where each piece is represented as a MIDI file.”
Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, methodology): “To test the ability of the model to generate polyphonic audio, we use a MIDI piano instrument playing a C chord with a gradually increasing number of notes.”
Neural Music Synthesis for Flexible Timbre Control (2018, methodology): “Our earlier experiments on continuous emission failed to stably perform autoregressive sampling due to teacher forcing, and the future work includes investigating this phenomenon comparing with [21], which used a mixture of logistics distributions to produce high-quality piano sounds.”
Source Coding of Audio Signals with a Generative Model (2020, methodology): “We trained the generative model using the Maestro dataset (v1) [9], which was divided into non-overlapping training, validation and test sets.”
LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training (2019, methodology): “It is, however, smaller than the 172 hours of piano music in the MAESTRO Dataset [3] used to train Music Transformer.”
Human-AI Co-Creation in Songwriting (2020, background): “In large part, these human-AI experiences were enabled by major advances in machine learning and deep generative models [18,66,68], many of which can now generate coherent pieces of long-form music [17,30,39,55].”
Fast and Flexible Neural Audio Synthesis (2019, methodology): “Wave2Midi2Wave [6] successfully conditioned a WaveNet to generate realistic piano audio based on a convolutional stack encoding a pianoroll.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, unknown): “Music Generation MuseNet [92], Jukedeck, WaveNet [93], AudioLM [94] Transformer, RNN, CNN MIDI Dataset, MAESTRO [95] ABC-notation, Music IS”
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models (2020, background): “Research has demonstrated the potential for modeling and synthesizing music, ranging from singlevoice sequences [13] and multi-part music [19, 34], to music with variable parts at each time step [4] and music with longterm structure over minutes [30, 37, 26].”
Musika! Fast Infinite Waveform Music Generation (2022, unknown): “We use the MAESTRO dataset [39], consisting of 200 hours of piano performances, to train a second-level au-toencoder and a latent GAN.”