A Framework for the Robust Evaluation of Sound Event Detection
By Ç. Bilen, Giacomo Ferroni, Francesco Tuveri, Juan Azcarreta, Sacha Krstulović. 2019
197 citations recorded by the source corpus.
This work defines a new framework for performance evaluation of polyphonic sound event detection (SED) systems, which overcomes the limitations of the conventional collar-based event decisions, event F-scores and event error rates. The proposed framework introduces a definition of event detection that is more robust against labelling subjectivity. It also resorts to polyphonic receiver operating characteristic (ROC) curves to deliver more global insight into system performance than F1-scores, and proposes a reduction of these curves into a single polyphonic sound detection score (PSDS), which allows system comparison independently from operating points (OPs). The presented method also delivers better insight into data biases and classification stability across sound classes. Furthermore, it can be tuned to varying applications in order to match a variety of user experience requirements. The benefits of the proposed approach are demonstrated by re-evaluating the baseline and two of the top-performing systems from DCASE 2019 Task 4.
Subfields: Computer Vision and Pattern Recognition, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Eduardo Fonseca
- Xavier Serra
- Xavier Favory
- Frederic Font
- Jordi Pons
- Romain Serizel
- Nicolas Turpault
- John R. Hershey
In Signal Processing
- Eduardo Fonseca
- Xavier Serra
- Frederic Font
- Xavier Favory
- Jordi Pons
- Scott Wisdom
- John R. Hershey
- Hakan Erdogan
Citations in the corpus
- FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “This trend has been adopted in other fields such as speaker recognition [102] and also recently in sound event detection (SED) [103].”
- Improving Sound Event Detection in Domestic Environments using Sound Separation (2020, background): “SED systems are also evaluated with poly-phonic sound event detection scores(PSDS) [27].”
- USM-SED - A Dataset for Polyphonic Sound Event Detection in Urban Sound Monitoring Scenarios (2021, methodology): “For evaluation purpose, both traditional SED metrics such as F-score or error rate [6] as well as recently proposed metrics such as the polyphonic sound detection score (PSDS) [7] are applicable.”