Appearance-and-Relation Networks for Video Classification

By Limin Wang, Wen Li, Luc Van Gool. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

376 citations recorded by the source corpus.

Spatiotemporal feature learning in videos is a fundamental problem in computer vision. This paper presents a new architecture, termed as Appearance-and-Relation Network (ARTNet), to learn video representation in an end-to-end manner. ARTNets are constructed by stacking multiple generic building blocks, called as SMART, whose goal is to simultaneously model appearance and relation from RGB input in a separate and explicit manner. Specifically, SMART blocks decouple the spatiotemporal learning module into an appearance branch for spatial modeling and a relation branch for temporal modeling. The appearance branch is implemented based on the linear combination of pixels or filter responses in each frame, while the relation branch is designed based on the multiplicative interactions between pixels or filter responses across multiple frames. We perform experiments on three action recognition benchmarks: Kinetics, UCF101, and HMDB51, demonstrating that SMART blocks obtain an evident improvement over 3D convolutions for spatiotemporal feature learning. Under the same training setting, ARTNets achieve superior performance on these three datasets to the existing state-of-the-art methods.1

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e6b534d8838f92461a478a3e737b73e08db94748

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Yi Yang
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Jitendra Malik
7. Jiashi Feng
8. Yu Qiao

Citations in the corpus

• StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2018, background): “Recently, there are many frameworks proposed to improve 3D convolution (Zolfaghari, Singh, and Brox 2018; Wang et al. 2017a; Xie et al. 2018; Tran et al. 2018; Wang et al. 2017b; Chen et al. 2018).”
• VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, methodology): “Since the introduction of Kinetics benchmarks [28], some supervised pre-trained models on it have been transferred to small-scale datasets for action recognition, such as 2D CNNs (TSN [71], TSM [40], TANet [48], TDN [70]), 3D CNNs (I3D [7], R(2+1)D [64], ARTNet [69], Slow-Fast [19]), Transformer (Ti”
• Video Classification With Channel-Separated Convolutional Networks (2019, background): “ARTNet [33] decouples spatial and temporal modeling into two parallel branches.”
• A Multigrid Method for Efficiently Training Video Models (2019, background): “Some recent studies propose lighter weight models that use efficient temporal modules [2,5,19,21,28–30,32,34,40,44,46] and/or exploit temporal redundancy [9,51].”
• TDN: Temporal Difference Networks for Efficient Action Recognition (2020, methodology): “In addition, some methods tried to leverage the idea of two stream network to design a multi-branch architecture to capture both appearance and motion or context information, with a carefully designed temporal module or two RGB inputs sampled at different FPS, including Non-local Net [39], ARTNet [3”
• Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, background): “A CTION recognition in videos has drawn enormous attention from the computer vision community in the past decades [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], partially due to its applications in many areas such as video surveillance, content-based analysis, and human-computer interaction.”
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, methodology): “The sparse sampling strategy (Wang et al. 2016) is utilized to extract T frames from the video clips ( T = 8 or 16 in our experiments, 8 by default without specific explanation).”
• ECO: Efficient Convolutional Network for Online Video Understanding (2018, methodology): “3D-Net: For the 3D network H3D we adopt several layers of 3D-Resnet18 [32], which is an efficient architecture used in many video classification works [32,15].”
• Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, neutral): “1 ARTNet [43] RGB+Flow ResNet-18 ImNet 72.”
• Learning Spatiotemporal Features via Video and Text Pair Discrimination (2020, background): “Since the breakthrough of AlexNet [30] in image recognition for representation learning, huge numbers of video-based deep models have been developed for action recognition [26,48,53,68,59,58,60,12].”
• When Video Classification Meets Incremental Classes (2021, background): “, a 2D spatial convolution and a 1D temporal convolution) [37, 45, 59], or mix 2D and 3D convolutions in a single network [48, 59, 65, 66].”
• Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “learning frameworks combining 2D CNNs and 3D CNNs have also been investigated for HAR [115], [116], [117].”
• X3D: Expanding Architectures for Efficient Video Recognition (2020, methodology): “For inference, we temporally max-pool prediction scores [3].”
• Temporal Shift Module for Efficient Video Understanding (2018, methodology): “TSN [42] and ARTNet [41] sample 25 frames from the image and apply 10 crops per frame leading to 250 processed frames per video.”