Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
By Nicolas Boulanger-Lewandowski, Yoshua Bengio, Pascal Vincent. 2012
739 citations recorded by the source corpus.
We investigate the problem of modeling symbolic sequences of polyphonic music in a completely general piano-roll representation. We introduce a probabilistic model based on distribution estimators conditioned on a recurrent neural network that is able to discover temporal dependencies in high-dimensional sequences. Our approach outperforms many traditional models of polyphonic music on a variety of realistic datasets. We show how our musical language model can serve as a symbolic prior to improve the accuracy of polyphonic transcription.
Subfields: Computer Vision and Pattern Recognition, Music, Representation & Self-Supervised Learning, Model Architectures, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Music
- Dong Yu
- Li Deng
- Simon E. Dixon
- Sander Dieleman
- François Pachet
- Gaëtan Hadjeres
- Kyunghyun Cho
- Emmanouil Benetos
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Ruslan Salakhutdinov
- Li Deng
- J. Urgen Schmidhuber
- Kyunghyun Cho
- Aaron C. Courville
- Quoc V. Le
- Geoffrey E. Hinton
In Model Architectures
- Yoshua Bengio
- Kyunghyun Cho
- Quoc V. Le
- Ruslan Salakhutdinov
- Aaron C. Courville
- Vladlen Koltun
- J. Urgen Schmidhuber
- Richard Socher
In Signal Processing
- Yoshua Bengio
- Yi-Hsuan Yang
- Ilya Sutskever
- Dong Yu
- Aaron C. Courville
- Jesse Engel
- Li Deng
- Kyunghyun Cho
Citations in the corpus
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, background): “We consider three polyphonic music datasets [see, e.g., Boulanger-Lewandowski et al., 2012] as well as two internal datasets provided by Ubisoft in which each sample is a raw speech representation.”
- Generating Sequences With Recurrent Neural Networks (2013, background): “Recurrent neural networks (RNNs) are a rich class of dynamic models that have been used to generate sequences in domains as diverse as music [6, 4], text [30] and motion capture data [29].”
- Neural Autoregressive Distribution Estimation (2016, methodology): “Larochelle and Lauly (2012); Zheng et al. (2015b) adapted NADE for topic modeling of documents and images, while Boulanger-Lewandowski et al. (2012) used NADE for modeling music sequential data.”
- Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “Other approaches have examined RNNs with richer expressivity, latent-variables for notes, or raw audio synthesis (Boulanger-Lewandowski et al., 2012; Gu et al., 2015; Chung et al., 2015).”
- On the difficulty of training recurrent neural networks (2012, methodology): “We address the task of polyphonic music prediction, using the datasets Piano-midi.de, Nottingham and MuseData described in Boulanger-Lewandowski et al. (2012) and language modelling at the character level on the Penn Treebank dataset (Mikolov et al. , 2012).”
- The challenge of realistic music generation: modelling raw audio at scale (2018, methodology): “More recently, deep learning and modern generative modelling techniques have been applied to this task [5, 7].”
- Representation Learning: A Review and New Perspectives (2012, methodology): “Another family of approaches, more in the spirit of Contrastive Divergence, relies on distinguishing positive examples (of the training distribution) and negative examples obtained by perturbations of the positive examples (Collobert and Weston, 2008; Bordes et al., 2012; Weston et al., 2010).”
- LSTM: A Search Space Odyssey (2015, methodology): “[41].”
- A Recurrent Latent Variable Model for Sequential Data (2015, background): “We argue, as have others [4, 2], that these complex dependencies cannot be modelled efficiently by the output probability models used in standard RNNs, which include either a simple unimodal distribution or a mixture of unimodal distributions.”
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “On Nottingham and JSB Chorales, the TCN with virtually no tuning outperforms the recurrent models by a considerable margin, and even outperforms some enhanced recurrent architectures for this task such as HF-RNN (Boulanger-Lewandowski et al., 2012) and Diago-nal RNN (Subakan & Smaragdis, 2017).”
- Deep Learning: Methods and Applications (2014, methodology): “The RNN has also been recently applied to music processing applications [22, 40, 41], where the use of ReLU hidden units instead of logistic or tanh nonlinearities are explored in the RNN.”
- Pre-training Graph Neural Networks with Kernels (2018, neutral): “[2012]; Boulanger-Lewandowski et al. [2012]) only involving input-to-hidden connections, i.e. assuming each item belonging to a sequence as independent of the others. This pre-training strategy is clearly unsatisfactory, because by definition the items belonging to the same sequence are dependent on”
- Sequential Neural Models with Stochastic Layers (2016, methodology): “Table 2 compares the average log-likelihood on the test sets obtained with SRNN and the models introduced in [3, 4, 13, 16].”
- Structured Inference Networks for Nonlinear State Space Models (2016, methodology): “3 (top and middle) depicts our results.”