Recurrent neural networks for polyphonic sound event detection in real life recordings

By Giambattista Parascandolo, Heikki Huttunen, Tuomas Virtanen. 2016

337 citations recorded by the source corpus.

In this paper we present an approach to polyphonic sound event detection in real life recordings based on bi-directional long short term memory (BLSTM) recurrent neural networks (RNNs). A single multilabel BLSTM RNN is trained to map acoustic features of a mixture signal consisting of sounds from multiple classes, to binary activity indicators of each event class. Our method is tested on a large database of real-life recordings, with 61 classes (e.g. music, car, speech) from 10 different everyday contexts. The proposed method outperforms previous approaches by a large margin, and the results are further improved using data augmentation techniques. Overall, our system reports an average F1-score of 65.5% on 1 second blocks and 64.7% on single frames, a relative improvement over previous state-of-the-art approach of 6.8% and 15.1% respectively.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “Main parameters for CRNN and VGG-like architectures are set via non-exhaustive preliminary experiments.”
CNN architectures for large-scale audio classification (2016, methodology): “More recent approaches use some form of DNN, including CNNs [12] and RNNs [13].”
Detection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge (2018, methodology): “Often based on mel-scale spectral representations of the signal for features, employed methods for sound event detection include HMMs [14], NMF [15]–[17], and recently a variety of temporally constrained deep learning methods such as convolutional neural networks (CNNs) [18]– [20] and long short-ter”
Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification (2016, methodology): “, [11], [23]), with Piczak [11] [which used random combinations oftime shifting, pitch shifting and time stretching for data augmentation] reporting that “simple aug-”
Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection (2017, neutral): “The main metric used in previous works [11], [14], [15] on TUT-SED 2009 dataset differs from the F1 score calculation”
Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks (2018, background): “CNN in this ﬁgure refers to a SELDnet architecture which had no RNN layers but just CNN and FC layers.”
Active Learning for Sound Event Detection (2020, background): “Previous sound event detection studies [29], [30] use synthetic datasets as primary evaluation datasets, since the timestamps of sound”
Audio-Visual Event Localization in Unconstrained Videos (2018, positive): “Approaches based on Hidden Markov Models (HMM), Gaussian Mixture Models (GMM), feed-forward Deep Neural Networks (DNN), and Bidirectional Long Short-Term Memory (BLSTM) [46] are developed in [10,23,36,43].”
Deep Learning for Audio Signal Processing (2019, methodology): “Alternatively, classifier architectures which model temporal information may be used: for example, recurrent neural networks may be applied to map a sequence of frame-wise acoustic features to a sequence of binary vectors representing event class activities [113].”
Vggsound: A Large-Scale Audio-Visual Dataset (2020, background): “More recently, the success of deep learning has motivated approaches based on CNNs [5, 27] or RNNs [28, 29, 30].”
Adaptive Pooling Operators for Weakly Labeled Sound Event Detection (2018, background): “The most recent research on SED is dominated by deep (fully connected) neural networks [27], convolutional networks [4], [28], [29], recurrent networks [30], [31], or convolutional-recurrent networks [32], [33].”