AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition

By Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, Rogério Schmidt Feris. 2021

70 citations recorded by the source corpus.

Temporal modelling is the key for efficient video action recognition. While understanding temporal information can improve recognition accuracy for dynamic actions, removing temporal redundancy and reusing past features can significantly save computation leading to efficient action recognition. In this paper, we introduce an adaptive temporal fusion network, called AdaFuse, that dynamically fuses channels from current and past feature maps for strong temporal modelling. Specifically, the necessary information from the historical convolution feature maps is fused with current pruned feature maps with the goal of improving both recognition accuracy and efficiency. In addition, we use a skipping operation to further reduce the computation cost of action recognition. Extensive experiments on Something V1&V2, Jester and Mini-Kinetics show that our approach can achieve about 40% computation savings with comparable accuracy to state-of-the-art methods. The project page can be found at https://mengyuest.github.io/AdaFuse/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “AdaFuse [182] selectively reuses certain feature channels from the previous”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “We directly quote the numbers re-ported in the published papers when possible or use authors provided source codes [60, 35] using the same backbone and experimental settings for a fair comparison.”
Adaptive Focus for Efficient Video Recognition (2021, background): “The average duration is 167 seconds; (3) Mini-Kinetics is a subset of Kinetics [24] introduced by [29, 30].”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “Motivated by this, we leverage the idea of dynamic inference [39, 37, 38, 61, 57], and adopt a policy network (referred to as multi-head interpreter) to decide which patches are uninformative and then discard them.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “Concurrent to our work, AdaFuse [34] dynamically fuses channels from current and history feature maps to reduce the computation.”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, background): “Some recent approaches [19, 23, 26] also explore dynamic temporal feature aggregation within 3D networks by fusing features along temporal axis controlled by gating functions conditioned on input clips, resulting in clear computational savings.”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “, Tiny Video Networks [39], channel-separated CNNs [49], and X3D [13]) or selecting salient frames/clips [61, 60, 30, 17, 57, 22, 34, 35, 37].”
VA-RED2: Video Adaptive Redundancy Reduction (2021, methodology): “Many adaptive computation methods have been recently proposed with the goal of improving efficiency (Bengio et al., 2015; 2013; Veit & Belongie, 2018; Wang et al., 2018; Graves, 2016; Meng et al., 2021).”