Multi-Fiber Networks for Video Recognition

By Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, Jiashi Feng. 2018

235 citations recorded by the source corpus.

In this paper, we aim to reduce the computational cost of spatio-temporal deep neural networks, making them run as fast as their 2D counterparts while preserving state-of-the-art accuracy on video recognition benchmarks. To this end, we present the novel Multi-Fiber architecture that slices a complex neural network into an ensemble of lightweight networks or fibers that run through the network. To facilitate information flow between fibers we further incorporate multiplexer modules and end up with an architecture that reduces the computational cost of 3D networks by an order of magnitude, while increasing recognition performance at the same time. Extensive experimental results show that our multi-fiber architecture significantly boosts the efficiency of existing convolution networks for both image and video recognition tasks, achieving state-of-the-art performance on UCF-101, HMDB-51 and Kinetics datasets. Our proposed model requires over 9\(\times \) and 13\(\times \) less computations than the I3D [1] and R(2+1)D [2] models, respectively, yet providing higher accuracy.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Other examples are ResNets [8] which are constructed by stacking residual blocks of fixed topology, ResNeXt models [9] which use multi-branch residual block modules, DenseNets [10] which use dense blocks as building blocks, or Multi-Fiber networks [11] which use parallel branches (“fibers”) connecte”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “However, we note that Transformers have been adopted on top of convolutional feature maps for action localization and recognition (Girdhar et al., 2019), video classification (Wang et al., 2018b; Chen et al., 2018b), and group activity recognition (Gavrilyuk et al., 2020).”
A2-Nets: Double Attention Networks (2018, methodology): “In the future, we want to explore integrating the double attention in recent compact network architectures [19, 16, 4], to leverage the expressiveness of the proposed method for smaller, mobile-friendly models.”
Central Similarity Quantization for Efficient Image and Video Retrieval (2019, methodology): “The “deep learning to hash” methods such as CNNH [44], DNNH [14], DHN [48], DCH [1] and Hash-Net [2] have been successfully applied to image hashing.”
Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution (2019, unknown): “The efficiency of Convolutional Neural Networks (CNNs) keeps increasing with recent efforts to reduce the inherent redundancy in dense model parameters [15, 31, 42] and in the channel dimension of feature maps [47, 18, 6, 9].”
Graph-Based Global Reasoning Networks (2018, methodology): “Video Action Recognition We run the baseline methods and our proposed method with the code released by [8] using PyTorch [27].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, background): “…29, 35] which extract spatial feature vectors from video frames and then integrate the obtained local feature sequence into one compact video descriptor for recognition; (2) the 2D [45, 36, 10, 7] or 3D convolution based [41, 11, 3, 52, 33, 47, 42, 44, 56, 5] end-to-end video classiﬁcation methods.”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, methodology): “06 22 9 MF-Net [39] RGB-3DConv 16 Verbs 44.”
VidTr: Video Transformer Without Convolutions (2021, methodology): “However, training 3D convnet from scratch was hard, initializing the 3D convnet weights by inﬂate from 2D networks was initially proposed in I3D [7] and soon proved applicable with different type of 2D network [10, 24, 58].”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, methodology): “There have already existed pretty much work on improving the efficiency of action recognition in videos [12, 35, 20, 2, 34, 27, 22, 28, 25, 19].”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, background): “group convolution [17, 30], have been explored in 3D models [5, 35, 34].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “explored, including mixed 2D and 3D networks [45], [46], grouped or depthwise 3D convolutions [24], [47], [48], [49], and the use of network architecture search [50].”
A Multigrid Method for Efficiently Training Video Models (2019, background): “Some recent studies propose lighter weight models that use efficient temporal modules [2,5,19,21,28–30,32,34,40,44,46] and/or exploit temporal redundancy [9,51].”
BEVT: BERT Pretraining of Video Transformers (2021, background): “3D CNNs are generally computationally expensive, and this motivates a line of research on efﬁcient video recognition [13, 22, 34, 51, 53, 68, 70].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Some work balances the accuracy-efficiency trade-off by using compressed video representations [68, 49] or designing efficient network architectures [72, 81, 10, 57].”
Video Classification With Channel-Separated Convolutional Networks (2019, methodology): “MFNet [5] ImageNet 72.”
X3D: Expanding Architectures for Efficient Video Recognition (2020, methodology): “MF-Net [9] 72.”
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2018, methodology): “Besides, we also implemented an ordinary 2-layered temporal Conv model (denoted as Ordinary Temporal-Conv) and RNN-based models (LSTM (Hochreiter and Schmidhuber 1997) and GRU (Cho et al. 2014)) for comparison.”