Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition

By Jinwoo Choi, Chen Gao, Joseph C.E. Messou, Jia-Bin Huang. 2019

214 citations recorded by the source corpus.

Human activities often occur in specific scene contexts, e.g., playing basketball on a basketball court. Training a model using existing video datasets thus inevitably captures and leverages such bias (instead of using the actual discriminative cues). The learned representation may not generalize well to new action classes or different tasks. In this paper, we propose to mitigate scene bias for video representation learning. Specifically, we augment the standard cross-entropy loss for action classification with 1) an adversarial loss for scene types and 2) a human mask confusion loss for videos where the human actors are masked out. These two losses encourage learning representations that are unable to predict the scene types and the correct actions when there is no evidence. We validate the effectiveness of our method by transferring our pre-trained model to three different tasks, including action classification, temporal localization, and spatio-temporal action detection. Our results show consistent improvement over the baseline model without debiasing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, background): “This can be explained by the strong scene bias exhibited in UCF101 [9].”
MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection (2021, background): “VAD depends on discriminative representations that clearly represent the events in a scene, while action recognition datasets pretrained feature encoders are not perfect for surveillance videos because of the existence of a domain gap [11, 3, 4].”
Learning De-biased Representations with Biased Representations (2019, background): “For instance, action recognition models rely heavily on static cues without learning temporal cues (Li et al., 2018; Li & Vasconcelos, 2019; Choi et al., 2019); we can regularise the 3D CNNs towards better generalisation across static biases by defining G to be the set of 2D CNNs.”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, unknown): “Video analysis [24] has been a prominent research topic in computer vision due to its vast potential applications, such as action recognition [32, 8], event detection [14], action localization [4, 49, 50], audio-visual scene analysis [16, 5, 12], etc .”