Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks

By Yu-Gang Jiang, Zuxuan Wu, Jun Wang, Xiangyang Xue, Shih-Fu Chang. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015

384 citations recorded by the source corpus.

In this paper, we study the challenging problem of categorizing videos according to high-level semantics such as the existence of a particular human action or a complex event. Although extensive efforts have been devoted in recent years, most existing works combined multiple video features using simple fusion strategies and neglected the utilization of inter-class semantic relationships. This paper proposes a novel unified framework that jointly exploits the feature relationships and the class relationships for improved categorization performance. Specifically, these two types of relationships are estimated and utilized by imposing regularizations in the learning process of a deep neural network (DNN). Through arming the DNN with better capability of harnessing both the feature and the class relationships, the proposed regularized DNN (rDNN) is more suitable for modeling video semantics. We show that rDNN produces better performance over several state-of-the-art approaches. Competitive results are reported on the well-known Hollywood2 and Columbia Consumer Video benchmarks. In addition, to stimulate future research on large scale video categorization, we collect and release a new benchmark dataset, called FCVID, which contains 91,223 Internet videos and 239 manually annotated categories.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1a0a01c78b746f44575f0c4f297e1621eb212dde

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “For example, FCVID contains some classes of static objects and scenes like “bridge” and “temple”, and thus we observe more computational savings than ACTIVITYNET and MINI-KINETICS, which are more activityfocused; on MINI-KINETICS, where categories are motionintensive, more computational resources ar”
Combining multiple sources of knowledge in deep CNNs for action recognition (2016, methodology): “Some fusion methods based on fully connected layers have been suggested to improve video classiﬁcation[9, 26].”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, methodology): “AdaFrame offers similar or better accuracies measured in mean average precision over the widely adopted uniform sampling strategy, a simple yet strong baseline, on FCVID and ACTIVITYNET respectively, while requiring 58.9% and 63.3% fewer computations on average, going as high as savings of 90.6%.”
Adaptive Focus for Efficient Video Recognition (2021, background): “We evaluate the effectiveness of AdaFocus on five video recognition benchmarks (i.e., ActivityNet, FCVID, MiniKinetics, Something-Something V1&V2).”
Black-box Adversarial Attacks on Video Recognition Models (2019, background): “ose to adapt image (2D) CNNs to video recognition. [1] explores various approaches for extending 2D CNNs to video recognition based on features extracted from individual frames. CNN+LSTM based models [30], [31] exploit the temporal information contained in successive frames, with recurrent layers ca”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “As mentioned, the dominant application domain so far has been action recognition, where the task is to predict a global action label for a given video (for example, [23, 12, 17, 11, 5]).”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “FCVID has 45, 611 videos for training and 45, 612 videos for testing across 239 classes.”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “To benchmark the effectiveness of our method, we use two challenging datasets namely ActivityNet [9] and FCVID (Fudan-Columbia Video Dataset) [13].”
Clean-Label Backdoor Attacks on Video Recognition Models (2020, background): “State-of-the-art video recognition models include the Inflated 3D ConvNet (I3D) [1] and the CNN+LSTM [34, 12].”
Human Action Recognition and Prediction: A Survey (2018, neutral): “FCVID (Jiang et al., 2018) 2015 91,233 – 239 – RGB Uncontrolled”
LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition (2019, positive): “For offline prediction, we compute average precision (AP) for each video category and use mean AP across all classes to measure the overall performance following [18, 12].”
Revisiting Video Saliency Prediction in the Deep Learning Era (2021, positive): “The list of the key terms is mainly built upon the labels of two datasets, MSCOCO [84] and FCVID [85], and is supplemented with about 40 extra keywords proposed by ourselves.”
Deep Multimodal Representation Learning: A Survey (2019, background): “[21], which impose a trace norm regularization over the network weights to reveal the hidden correlations and diversity of the multimodal features.”
Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “Dataset Year Modality #Class #Subject #Sample #View KTH [347] 2004 RGB 6 25 2,391 1 Weizmann [40] 2005 RGB 10 9 90 1 IXMAS [348] 2006 RGB 11 10 330 5 HDM05 [349] 2007 RGB,S 130 5 2,337 1 Hollywood [350] 2008 RGB 8 - 430 Hollywood2 [351] 2009 RGB 12 - 3,669 MSR-Action3D [217] 2010 S,D 20 10 567 1 Oly”
EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition (2018, background): “[21] introduce a unified deep learning framework that jointly exploits feature and class relationships for action recog-”
Blind late fusion in multimedia event retrieval (2016, methodology): “This strategy is used, because in the paper published with this data set [5], the training of the SVM had a better performance compared to using the DCNN.”