Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

By Sharath Adavanne, Archontis Politis, Joonas Nikunen, Tuomas Virtanen. IEEE Journal on Selected Topics in Signal Processing, 2018

599 citations recorded by the source corpus.

In this paper, we propose a convolutional recurrent neural network for joint sound event localization and detection (SELD) of multiple overlapping sound events in three-dimensional (3-D) space. The proposed network takes a sequence of consecutive spectrogram time frames as input and maps it to two outputs in parallel. As the first output, the sound event detection (SED) is performed as a multi-label classification task on each time frame producing temporal activity for all the sound event classes. As the second output, localization is performed by estimating the 3-D Cartesian coordinates of the direction-of-arrival (DOA) for each sound event class using multi-output regression. The proposed method is able to associate multiple DOAs with respective sound event labels and further track this association with respect to time. The proposed method uses separately the phase and magnitude component of the spectrogram calculated on each audio channel as the feature, thereby avoiding any method- and array-specific feature extraction. The method is evaluated on five Ambisonic and two circular array format datasets with different overlapping sound events in anechoic, reverberant, and real-life scenarios. The proposed method is compared with two SED, three DOA estimation, and one SELD baselines. The results show that the proposed method is generic and applicable to any array structures, robust to unseen DOA values, reverberation, and low SNR scenarios. The proposed method achieved a consistently higher recall of the estimated number of DOAs across datasets in comparison to the best baseline. Additionally, this recall was observed to be significantly better than the best baseline method for a higher number of overlapping sound events.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/4fc3d7659eb095788d6e02ae955adb1d1350f46d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

A multi-room reverberant dataset for sound event localization and detection (2019, methodology): “As the benchmark, we provide a modified version of the recently proposed SELDnet(3) [5] and report the results on the multi-room reverberant dataset.”
A Hybrid Parametric-Deep Learning Approach for Sound Event Localization and Detection (2019, methodology): “This is probably due to the complexity added by the association step [6], and its lack of robustness under highly reverberant scenarios.”
Active Learning for Sound Event Detection (2020, methodology): “The second dataset is TAU Spatial Sound Events 2019 - Ambisonic, which is used in the challenge of DCASE 2019 [28], as task 3.”
Machine learning in acoustics: Theory and applications. (2019, methodology): “107 the event detection and DOA estimation tasks are combined into a signal DNN architecture based on convolutional RNNs.”
A Survey of Sound Source Localization with Deep Learning Methods (2021, methodology): “The same baseline neural architecture of Adavanne et al. [2019a] was used by Kapka and Lewandowski [2019], with one separate (but identical, except for the output layer) CRNN instance for each subtask: source counting (up to two sources), DoA estimation of source 1 (if applicable), DoA estimation of”