The music information retrieval evaluation exchange (2005-2007): A window into music information retrieval research

By J. Stephen Downie. Acoustical Science and Technology, 2008

329 citations recorded by the source corpus.

The Music Information Retrieval Evaluation eXchange (MIREX) is the community-based framework for the formal evaluation of Music Information Retrieval (MIR) systems and algorithms. By looking at the background, structure, challenges, and contributions of MIREX this paper provides some insights into the world of MIR research. Because MIREX tasks are defined by the community they reflect the interests, techniques, and research paradigms of the community as a whole. Both MIREX and MIR have a strong bias toward audio-based approaches as most MIR researchers have strengths in signal processing. Spectral-based approaches to MIR tasks have led to advancements in the MIR field but they now appear to be reaching their limits of effectiveness. This limitation is called the “glass ceiling” problem and the MIREX results data support its existence. The post-hoc analyses of MIREX results data indicate that there are groups of systems that perform equally well within various MIR tasks. There are many challenges facing MIREX and MIR research most of which have their root causes in the intellectual property issues surrounding music. The current inability of researchers to test their approaches against the MIREX test collections outside the annual MIREX cycle is hindering the rapid development of improved MIR systems.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/90cd005a097750cdb020cafa0470ab3d41219afb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

State of the Art Report: Music Emotion Recognition: A State of the Art Review (2010, unknown): “The annual Music Information Research Evaluation eXchange (MIREX) is a community-based framework for formally evaluating Music-IR systems and algorithms [2], which included audio music mood classification as a task for the first time in 2007 [3].”
Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching (2016, methodology): “Given a MIDI transcription and human-generated ground truth data, we can extract corresponding information from the MIDI file and compare using the evaluation metrics employed in the Music Information Retrieval Evaluation eXchange (MIREX) (Downie, 2008).”
MedleyDB: A Multitrack Dataset for Annotation-Intensive MIR Research (2014, methodology): “They found that the evaluation is highly sensitive to systematic annotation errors, that performance on excerpts is not necessarily a good predictor for performance on full songs, and that the collections used for the MIREX evaluation [5] are too small for the results to be statistically stable.”
MIR_EVAL: A Transparent Implementation of Common MIR Metrics (2014, methodology): “For the past 10 years, the yearly Music Information Retrieval Evaluation eXchange (MIREX) has been a forum for comparing MIR algorithms over common datasets [6].”
Melody Extraction From Polyphonic Music Signals Using Pitch Contour Characteristics (2012, methodology): “The described system was submitted to the 2010 and 2011 Music Information Retrieval Evaluation eXchange (MIREX), an annual campaign in which different state-of-the-art MIR algorithms are evaluated against the same data-sets in order to compare their performance [31].”
Detection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge (2018, background): “[2] has been running yearly for over a decade already.”
Exploring automatic music annotation with "acoustically-objective" tags (2010, methodology): “More specifically, they were incorporated into each of the top performing autotagging systems in the 2008 MIREX tag classification task [1, 3, 13, 21].”
Fundamentals of Music Processing (2015, methodology): “Such systems were already proposed some decades ago (see [10, 57] and [12] for an overview).”
Evaluation of Algorithms Using Games: The Case of Music Tagging (2009, methodology): “This need is reﬂected in the recent addition of the audio tag classi-ﬁcation (ATC) task at MIREX 2008, and the introduction of new music tagging algorithms [1, 2].”