Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection

By Emre Çakir, Giambattista Parascandolo, Toni Heittola, Heikki Huttunen, Tuomas Virtanen. 2017

616 citations recorded by the source corpus.

Sound events often occur in unstructured environments where they exhibit wide variations in their frequency content and temporal structure. Convolutional neural networks (CNNs) are able to extract higher level features that are invariant to local spectral and temporal variations. Recurrent neural networks (RNNs) are powerful in learning the longer term temporal context in the audio signals. CNNs and RNNs as classifiers have recently shown improved performances over established methods in various sound recognition tasks. We combine these two approaches in a convolutional recurrent neural network (CRNN) and apply it on a polyphonic sound event detection task. We compare the performance of the proposed CRNN method with CNN, RNN, and other established methods, and observe a considerable improvement for four different datasets consisting of everyday sound events.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “h as in different types of speech (either spoken, sung, screamed, yelled or whispered). This is interesting as CRNNs were originally proposed for speech recognition [124] before being adapted for SER [20]. Other exceptions include some human sounds and animal vocalizations of marked temporal behavio”
Unsupervised Learning of Semantic Audio Representations (2017, background): “The last few years have seen great advances in nonspeech audio processing, as popular deep learning architectures developed in the speech and image processing communities have been ported to this relatively understudied domain [1, 2, 3, 4].”
A Hybrid Parametric-Deep Learning Approach for Sound Event Localization and Detection (2019, methodology): “We conducted a set of preliminary experiments with different types of networks including a VGG-like net, a less deep CNN [19], a Mobilenetv1 [20] and a CRNN [21].”
An Active Learning Method Using Clustering and Committee-Based Sample Selection for Sound Event Classification (2018, background): “Nowadays, the majority of sound event classification systems [6, 7] are based on supervised learning, which depends on annotated recordings as training material.”
Scaper: A library for soundscape synthesis and augmentation (2017, methodology): “The second largest, TUT-SED synthetic 2016 (also synthesized) [26], is roughly 9.”
Deep Learning for Audio Signal Processing (2019, methodology): “al information may be used: for example, recurrent neural networks may be applied to map a sequence of frame-wise acoustic features to a sequence of binary vectors representing event class activities [113]. Similarly to other supervised learning tasks, convolutional neural networks can be highly eff”
Detection and Classification of Acoustic Scenes and Events: Outcome of the DCASE 2016 Challenge (2018, methodology): “[20] for labelc.”
Automatic acoustic detection of birds through deep learning: The first Bird Audio Detection challenge (2018, methodology): “In order to ensure methods could work in conditions different from those in the training data, various participants explored self-adaptation, in which a trained network is fine-tuned upon exposure to the new conditions (without needing any additional ground-truth information) (Grill and Schlüter, 20”
Adaptive Pooling Operators for Weakly Labeled Sound Event Detection (2018, background): “The most recent research on SED is dominated by deep (fully connected) neural networks [27], convolutional networks [4, 28, 29], recurrent networks [30, 31], or convolutional-recurrent networks [32, 33].”
Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks (2018, result): “Similar performance drop on real-life datasets has also been reported on SED datasets in other studies [37].”
Computational bioacoustics with deep learning: a review and roadmap (2021, methodology): “(2017) which represents the maturing of audio recognition based on convolutional neural networks (CNNs)—it introduced a dataset (AudioSet) and a NN architecture (VGGish) both now widely-used; and convolutional-recurrent neural network (CRNN) methods (Çakır et al., 2017).”
Active Learning for Sound Event Detection (2020, background): “Previous sound event detection studies [29], [30] use synthetic datasets as primary evaluation datasets, since the timestamps of sound”
A Deep Residual Network for Large-Scale Acoustic Scene Analysis (2019, methodology): “In this work, we focus on deep convolutional neural networks (CNNs) for audio classification due to the success in many prior studies [14, 15, 16, 17, 18, 19].”
DCASE2017 Challenge Setup: Tasks, Datasets and Baseline System (2017, methodology): “The most used features for sound event detection are mel-scale representations, namely cepstral coefficients or log energies [12, 13, 14], and they are used with various machine learning methods, including HMMs [12], non-negative matrix factorization (NMF) [15, 16], random forests [11], and DNNs [14”
Semi-supervised Acoustic Event Detection Based on Tri-training (2019, methodology): “Our experimented models include ResNet [24], DenseNet and Conv-RNN [25, 26] with different layers, which are among the state-of-the-art models for acoustic event detection.”
Machine learning in acoustics: Theory and applications. (2019, background): “105, convolutional recurrent NNs achieve state-of-the art results in the sound event detection task in the 2017 Detection and Classiﬁcation of Acoustic Scenes and Events (DCASE) challenge.”
VOICe: A Sound Event Detection Dataset For Generalizable Domain Adaptation (2019, methodology): “For example, in the DCASE workshop and challenge2, the recent state-of-the-art results are obtained by applying stacked convolutional and recurrent neural networks (CRNNs) [2, 8].”
The Impact of Missing Labels and Overlapping Sound Events on Multi-label Multi-instance Learning for Sound Event Classification (2019, methodology): “Particularly convolutional neural networks (CNN), potentially with some recurrent layers added, have shown good performance with respect to the considered task [5, 7].”
Sound Event Detection Using Point-Labeled Data (2019, methodology): “Deep neural networks have been successfully applied to recent SED systems [4, 5, 6] and are the current state-of-the-art.”