Audio-based multimedia event detection using deep recurrent neural networks

By Yun Wang, Leonardo Neves, Florian Metze. 2016

74 citations recorded by the source corpus.

Multimedia event detection (MED) is the task of detecting given events (e.g. birthday party, making a sandwich) in a large collection of video clips. While visual features and automatic speech recognition typically provide the best features for this task, nonspeech audio can also contribute useful information, such as crowds cheering, engine noises, or animal sounds. MED is typically formulated as a two-stage process: the first stage generates clip-level feature representations, often by aggregating frame-level features; the second stage performs binary or multi-class classification to decide whether a given event occurs in a video clip. Both stages are usually performed "statically", i.e. using only local temporal information, or bag-of-words models. In this paper, we introduce longer-range temporal information with deep recurrent neural networks (RNNs) for both stages. We classify each audio frame among a set of semantic units called "noisemes" the sequence of frame-level confidence distributions is used as a variable-length clip-level representation. Such confidence vector sequences are then fed into long short-term memory (LSTM) networks for clip-level classification. We observe improvements in both frame-level and clip-level performance compared to SVM and feed-forward neural network baselines.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “1 SER has gained increasing attention in the past few years, becoming a key component in applications related to healthcare [1]–[3], urban sound planning [4], bioacoustics monitoring [5]–[7], multimedia event detection [8] and large-scale event discovery [9], surveillance [10, 11], or noise monitori”
Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection (2017, unknown): “SED has been drawing a surging amount of interest in recent years with applications including audio surveillance [1], healthcare monitoring [2], urban sound analysis [3], multimedia event detection [4] and bird call detection [5].”
Multimodal Speech Emotion Recognition Using Audio and Text (2018, methodology): “Motivated by the architecture used in [25, 26], we build an audio recurrent encoder (ARE) to predict the class of a given audio signal.”