Onsets and Frames: Dual-Objective Piano Transcription

By Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, Douglas Eck. International Society for Music Information Retrieval Conference, 2017

335 citations recorded by the source corpus.

We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames. Our model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note to start unless the onset detector also agrees that an onset for that pitch is present in the frame. We focus on improving onsets and offsets together instead of either in isolation as we believe this correlates better with human musical perception. Our approach results in over a 100% relative improvement in note F1 score (with offsets) on the MAPS dataset. Furthermore, we extend the model to predict relative velocities of normalized audio which results in more natural-sounding transcriptions.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/b2c184a5f3899ddae4edb9ae3fbc9dee53a5f2a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

General-purpose, long-context autoregressive modeling with Perceiver AR (2022, methodology): “ation of the Onsets and Frames model (Hawthorne et al., 2018).”
Neural Music Synthesis for Flexible Timbre Control (2018, methodology): “This input representation is inspired by [22] which showed that jointly training on onsets and frames performs better than using frame information only; similarly, we want the network to maximally utilize the onsets which have the most relevant information on the attack sounds, while still receiving”
PIANOTREE VAE: Structured Representation Learning for Polyphonic Music (2020, methodology): “The objective evaluation is performed by comparing different models in terms of their reconstruction accuracy of pitch onsets and note duration [39, 40], which are commonly used measurements in music information retrieval tasks.”
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset (2018, methodology): “Encoder, P ( notes | audio ) : An Onsets and Frames (Hawthorne et al., 2018) transcription model to produce a symbolic representation (MIDI) from raw audio.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, positive): “Hsiao et al. further employed Onsets and Frames piano transcription (Hawthorne et al., 2018), madmom beat tracking tool (B¨ock et al., 2016), and chorder rule-based chord detection 10 to transcribe the audio into MIDI format with tempo, beat, and chord information.”
Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions (2020, methodology): “ormationretrieval(MIR) techniques such as downbeat tracking [2, 3, 16] and chord recognition [17, 35], which differentiates it from existing approaches. Moreover, we use automatic music transcription [1, 21, 25] to prepare the MIDI data, so the model learns from expressive piano performances. We sho”
PolyRNN: A time-resolved model of polyphonic musical expectations aligned with human brain responses (2025, unknown): “We then converted the audio files into MIDI files with the Onsets and Frames Transcription tool (Hawthorne et al., 2017).”