Metrics for Polyphonic Sound Event Detection

By Annamaria Mesaros, Toni Heittola, Tuomas Virtanen. 2016

649 citations recorded by the source corpus.

This paper presents and discusses various metrics proposed for evaluation of polyphonic sound event detection systems used in realistic situations where there are typically multiple sound sources active simultaneously. The system output in this case contains overlapping events, marked as multiple sounds detected as being active at the same time. The polyphonic system output requires a suitable procedure for evaluation against a reference. Metrics from neighboring fields such as speech recognition and speaker diarization can be used, but they need to be partially redefined to deal with the overlapping events. We present a review of the most common metrics in the field and the way they are adapted and interpreted in the polyphonic case. We discuss segment-based and event-based definitions of each metric and explain the consequences of instance-based and class-based averaging using a case study. In parallel, we provide a toolbox containing implementations of presented metrics.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/22b76dd07b708cf091daad62e704a7a4342b8192

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Detection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge (2018, methodology): “The metrics are presented in detail in [53].”
A Deep Residual Network for Large-Scale Acoustic Scene Analysis (2019, background): “A labeling is determined to be a hit if both the starting and end times are within 200ms of the groundtruth, and that the event duration is no less than 80% of the true event duration [30].”
DCASE2017 Challenge Setup: Tasks, Datasets and Baseline System (2017, methodology): “The four cross-validation folds are treated as a single experiment: the metrics are calculated by accumulating error counts (insertions, deletions, substitutions) over all folds [27], not by averaging the individual folds nor the individual class performance.”
Scaper: A library for soundscape synthesis and augmentation (2017, methodology): “Both models are implemented in Keras [17] and trained using the Adam optimizer [29] with binary cross-entropy loss over 300 epochs with an early-stopping criterion of 100 epochs with no improvement to the segment-based F-measure [30], computed at a 1 s temporal resolution5 using sed eval [30] where ”
Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection (2017, neutral): “The second evaluation metric is segment-based error rate as proposed in [41].”
Active Learning for Sound Event Detection (2020, methodology): “In this study, a segment-based error rate (ER) is used to evaluate the performance of a SED model [31].”
A Framework for the Robust Evaluation of Sound Event Detection (2019, methodology): “However, segment-based evaluation such as in [9, 10], as well as the evaluation of monophonic systems, can be derived as special cases.”
Open-Source Practices for Music Signal Processing Research: Recommendations for Transparent, Sustainable, and Reproducible Audio Research (2019, methodology): “However, for domainspecific MIR problems, software packages have been developed to standardize evaluations, such as mir eval for music description and source separation [4], sed eval for sound event detection [30], and rival for recommender systems [31].”
A multi-room reverberant dataset for sound event localization and detection (2019, methodology): “For SED, we use the F-score and error rate (ER) calculated in one-second segments [8].”
Sound event detection in synthetic audio: Analysis of the dcase 2016 task results (2017, methodology): “Among the four metrics considered in the DCASE 2016 challenge [4], the class-wise event-based F-measure (Fcweb) is considered in this paper for providing a statistical analysis of the results.”
Deep Learning for Audio Signal Processing (2019, background): “In event detection, performance is typically measured using equal error rate or F-score, where the true positives, false positives and false negatives are calculated either in fixed-length segments or per event [84], [85].”
Sound Event Detection in Synthetic Domestic Environments (2020, positive): “The metrics are computed using the sed eval library [18].”
Robust Laughter Detection in Noisy Environments (2021, methodology): “…applications are concerned with capturing the entirety of a laughter event rather than just counting the total number of events, we use segment-based metrics [15], with segments deﬁned in this case as individual frames lasting 23 milliseconds, to calculate precision, recall, and F1 scores.”
Computational bioacoustics with deep learning: a review and roadmap (2021, background): “…datasets are usually “unbalanced”, having many more items of one category than another, it is common to account for this—for example by using macro-averaging , calculating performance for each class and then taking the average of those to give equal weight to each class (Mesaros et al., 2016).”
TUT database for acoustic scene classification and sound event detection (2016, background): “Evaluation of system performance for sound event detection uses error rate and F-score in a fixed time grid, as defined in [20].”
Improving Sound Event Detection in Domestic Environments using Sound Separation (2020, positive): “5) using the sed eval library [26].”
Sound Event Detection Using Point-Labeled Data (2019, methodology): “More details about the metrics for SED can be found in [16].”
Sound Event Detection in Domestic Environments with Weakly Labeled Data and Soundscape Synthesis (2019, methodology): “The metrics are computed using the sed eval library [23].”