Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

By Peng Wu, Yujiao Shi, Yujia Sun, Fang Shao, Zhaoyang Wu, Zhiwei Yang. European Conference on Computer Vision, 2020

543 citations recorded by the source corpus.

Violence detection has been studied in computer vision for years. However, previous work are either superficial, e.g., classification of short-clips, and the single scenario, or undersupplied, e.g., the single modality, and hand-crafted features based multimodality. To address this problem, in this work we first release a large-scale and multi-scene dataset named XD-Violence with a total duration of 217 hours, containing 4754 untrimmed videos with audio signals and weak labels. Then we propose a neural network containing three parallel branches to capture different relations among video snippets and integrate features, where holistic branch captures long-range dependencies using similarity prior, localized branch captures local positional relation using proximity prior, and score branch dynamically captures the closeness of predicted score. Besides, our method also includes an approximator to meet the needs of online detection. Our method outperforms other state-of-the-art methods on our released dataset and other existing benchmark. Moreover, extensive experimental results also show the positive effect of multimodal (audio-visual) input and modeling relationships. The code and dataset will be released in this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8f28873be3601c5a2736996eba543cf51950a381

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, methodology): “To this end, we introduce a multiple instance learning (MIL) based mechanism inspired by [32, 43, 44], which can be presented as follows,”
Self-Training Multi-Sequence Learning with Transformer for Weakly Supervised Video Anomaly Detection (2022, methodology): “Following previous works (Wu et al. 2020; Tian et al. 2021), we use the AP (Average Precision) as our metric for XD-Violence.”
Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning (2021, methodology): “The learning rate is set to 0.001 for ShanghaiTech and UCF-Crime, and 0.0001 for XD-Violence.”