This time with feeling: learning expressive musical performance

By Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, Karen Simonyan. 2018

258 citations recorded by the source corpus.

Music generation has generally been focused on either creating scores or interpreting them. We discuss differences between these two problems and propose that, in fact, it may be valuable to work in the space of direct performance generation: jointly predicting the notes and also their expressive timing and dynamics. We consider the significance and qualities of the dataset needed for this. Having identified both a problem domain and characteristics of an appropriate dataset, we show an LSTM-based recurrent network model that subjectively performs quite well on this task. Critically, we provide generated examples. We also include feedback from professional composers and musicians about some of these examples.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

Music Transformer: Generating Music with Long-Term Structure (2018, methodology): “Instead, the polyphonic performance is serialized into a sequence of one hot encoded events as proposed in (Oore et al., 2018).”
Encoding Musical Style with Transformer Autoencoders (2019, methodology): “The performances are encoded using the vocabulary as described in (Oore et al., 2018), which captures expressive dynamics and timing.”
Pop Music Transformer: Generating Music with Rhythm and Harmony (2020, methodology): “To convert a score into a sequence, [ Oore et al. , 2018 ] uses additionally the T IME -”
Variable-rate discrete representation learning (2021, background): “They have signiﬁcantly advanced the state of the art since (Simon & Oore, 2017; Huang et al., 2019; Payne, 2019; Oore et al., 2020).”
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “As discussed in [30], a score to be converted into events can be either a MIDI score with no expressive dynamics and timing, or a MIDI performance that has been converted from an expressive audio recording by means of, for example, a MIDI keyboard, or automatic music transcription [1].”
PolyRNN: A time-resolved model of polyphonic musical expectations aligned with human brain responses (2025, unknown): “( where is the conditional probability of the pitch or timing of the n th event of the sequence. long short-term memory (LSTM) layers (see Oore et al., 2020 for details).”