MIR_EVAL: A Transparent Implementation of Common MIR Metrics

By Colin Raffel, Brian McFee, Eric J. Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, D. Ellis. International Society for Music Information Retrieval Conference, 2014

639 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/6d37fbd2fccaef3ecdf75d34a7aee18ab9519a6f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Kristen Grauman
2. Yoshua Bengio
3. Ruohan Gao
4. Chuang Gan
5. Zhe Gan
6. Dong Yu
7. Andrea Vedaldi
8. Andrew Zisserman

Citations in the corpus

• Invariances and Data Augmentation for Supervised Music Transcription (2017, methodology): “Average Precision is computed by scikit-learn [23]; Accuracy and Error use mir eval [24].”
• Crepe: A Convolutional Representation for Pitch Estimation (2018, methodology): “We use the reference implementation provided in mir eval [27] to compute the evaluation metrics.”
• Co-Separating Sounds of Visual Objects (2019, methodology): “We use the widely used mir eval library [35] to evaluate the source separation and report the standard metrics: Signal-to-Distortion Ration (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifact Ratio (SAR).”
• SDR – Half-baked or Well Done? (2018, methodology): “The same issues apply to an early Python version of BSS eval, bss eval(2) [18].”
• Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching (2016, methodology): “We utilized the evaluation library mir_eval (Raffel et al., 2014) to compute this score.”
• Open-Source Practices for Music Signal Processing Research: Recommendations for Transparent, Sustainable, and Reproducible Audio Research (2019, methodology): “Confounding factors can arise from quirks in data collection [2], subtle design choices in feature representations [3], or unstated assumptions in the evaluation criteria [4].”
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “We evaluate the speech separation results using a series of standard metrics including Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signalto-Artifacts Ratio (SAR) from the mir eval library [61].”
• SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures (2019, methodology): “evaluated using the signal-to-distortion ratio (SDR) (as defined by [55] and computed using [56]) or the frequency-weighted signal-to-noise ratio (fw-SNR) computed using tools provided”
• 2.5D Visual Sound (2018, methodology): “To evaluate source separation quality, we use the widely used mir eval library [36], and the standard metrics: Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifact Ratio (SAR).”
• The Sound of Pixels (2018, methodology): “The results reported in this paper were obtained by using the open-source mir eval [47] library.”
• Robust sound event detection in bioacoustic sensor networks (2019, methodology): “To ensure optimum matching of detected peaks to reference peaks while ensuring each reference peak can only be matched to a single estimated peak, we treat the problem as a maximum bipartite graph matching problem [98] and use the implementation provided in the mir_eval Python library for efficiency”
• Self-supervised Audio-visual Co-segmentation (2019, methodology): “The Signal to Distortion Ratio (SDR) and the Signal to Interference Ratio (SIR) are two commonly used sound source separation metrics [26], and were calculated using the mir-eval library [27].”
• Onsets and Frames: Dual-Objective Piano Transcription (2017, methodology): “We use the mir eval library [16] to calculate notebased precision, recall, and F1 scores.”
• Singing Voice Separation with Deep U-Net Convolutional Networks (2017, methodology): “We compute performance measures using the mir eval toolkit [22].”
• librosa: Audio and Music Signal Analysis in Python (2015, methodology): “The resulting outputs are designed to work with other common MIR tools, such as mir_eval [Raffel14] and sonic-visualiser [Cannam10].”
• MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training (2023, methodology): “The model is evaluated using the f measure implemented in mir eval (Raffel et al., 2014), and the prediction is considered correct if the difference between the predicted event and the ground truth does not exceed 20ms.”
• M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus (2022, methodology): “We measure all the scores using utilities in the mir_eval library [34] with onset tolerance set to 100ms.”