VA-RED2: Video Adaptive Redundancy Reduction

By Bowen Pan, Rameswar Panda, Camilo Luciano Fosco, Chung-Ching Lin, Alex Andonian, Yue Meng, Kate Saenko, Aude Oliva, Rogério Schmidt Feris. 2021

22 citations recorded by the source corpus.

Performing inference on deep learning models for videos remains a challenge due to the large amount of computational resources required to achieve robust recognition. An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both. The type of redundant features depends on the dynamics and type of events in the video: static videos have more temporal redundancy while videos focusing on objects tend to have more channel redundancy. Here we present a redundancy reduction framework, termed VA-RED$^2$, which is input-dependent. Specifically, our VA-RED$^2$ framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions. To keep the capacity of the original model, after fully computing the necessary features, we reconstruct the remaining redundant features from those using cheap linear operations. We learn the adaptive policy jointly with the network weights in a differentiable way with a shared-weight mechanism, making it highly efficient. Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves $20\% - 40\%$ reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss. Project page: http://people.csail.mit.edu/bpan/va-red/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “neously has attracted researches in both static [193], [194] and dynamic networks [195], [196], [197], [198].”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, background): “Some recent approaches [19, 23, 26] also explore dynamic temporal feature aggregation within 3D networks by fusing features along temporal axis controlled by gating functions conditioned on input clips, resulting in clear computational savings.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “Many conditional computation methods have been recently proposed with the goal of improving computational efﬁciency (Bengio et al., 2015; 2013; Veit & Belongie, 2018; Wang et al., 2018b; Graves, 2016; Meng et al., 2020; Pan et al., 2021).”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “Motivated by this, we leverage the idea of dynamic inference [39, 37, 38, 61, 57], and adopt a policy network (referred to as multi-head interpreter) to decide which patches are uninformative and then discard them.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Similar in spirit, dynamic methods for efﬁcient video recognition adaptively select salient frames/clips [64, 61, 30, 9, 58, 23], utilize audio [14], reduce feature redundancy [38], or select frame resolutions [60, 34].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “, Tiny Video Networks [39], channel-separated CNNs [49], and X3D [13]) or selecting salient frames/clips [61, 60, 30, 17, 57, 22, 34, 35, 37].”
Adaptive Focus for Efficient Video Recognition (2021, background): “Recent research efforts have been made towards improving the efficiency of video recognition, via designing light-weighted architectures [38, 46, 32, 37, 54, 27] or perform dynamic computation on a per-video basis [49, 45, 25, 52, 26, 30].”