LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition

By Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, Larry S. Davis. 2019

116 citations recorded by the source corpus.

This paper presents LiteEval, a simple yet effective coarse-to-fine framework for resource efficient video recognition, suitable for both online and offline scenarios. Exploiting decent yet computationally efficient features derived at a coarse scale with a lightweight CNN model, LiteEval dynamically decides on-the-fly whether to compute more powerful features for incoming video frames at a finer scale to obtain more details. This is achieved by a coarse LSTM and a fine LSTM operating cooperatively, as well as a conditional gating module to learn when to allocate more computation. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet, and the results demonstrate LiteEval requires substantially less computation while offering excellent classification accuracy for both online and offline predictions.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “To reduce redundant computation at each time step, LiteEval [176] makes a choice between two LSTM models with different computational costs.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “In particular, existing methods for CNNs have explored altering input samples [28, 29, 39, 50, 51, 55], skipping network layers [10, 18, 41, 45, 52] and channels [1, 21], early exiting with a multi-classifier structure [2,17,19], to name a few.”
BEVT: BERT Pretraining of Video Transformers (2021, background): “There is a plethora of work on video understanding with CNNs, most of which focus on learning spatial-temporal features [9, 22, 23, 34, 50, 53, 57, 61, 62].”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “At each time step, LiteEval computes coarse features with a lightweight CNN to determine whether to examine the current frame more carefully using a heavy CNN. Meng et al. [33] propose to adaptively select the input resolution, on a per-frame basis to further balance the accuracy vs compute.”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, methodology): “[35] proposed a coarse-tofine framework automatically adjusting and selecting the suitable network structure for feature extraction from input data, which achieves good trade-off between computational cost and accuracy.”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, result): “Table 5 shows AdaShare is better than AdaShare-RL in each task and overall performance, in line with the comparison in [63].”
Adaptive Focus for Efficient Video Recognition (2021, methodology): “Recent research efforts have been made towards improving the efficiency of video recognition, via designing light-weighted architectures [38, 46, 32, 37, 54, 27] or perform dynamic computation on a per-video basis [49, 45, 25, 52, 26, 30].”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Similar in spirit, dynamic methods for efficient video recognition adaptively select salient frames/clips [63, 60, 30, 9, 57, 23], utilize audio [14], reduce feature redundancy [38], or select frame resolutions [59, 34].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “82% mAP improvement over LiteEval [59] with similar GFLOPs on ActivityNet and FCVID respectively.”
Human Action Recognition From Various Data Modalities: A Review (2020, unknown): “[73] leveraged two LSTMs, which operate on coarse-scale and fine-scale CNN features cooperatively for efficient HAR.”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “• LiteEval [61]: uses MobileNetV2/ResNet-101 as Policy Network/backbone.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “To illustrate the efﬁciency of our framework, we compute the ﬂoating point operations (FLOPS), which is a hardware-independent metric and widely used in the ﬁeld of efﬁcient action recognition 1 (Wu et al., 2019b; Gao et al., 2020; Meng et al., 2020; Fan et al., 2019).”
3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “By integrating SGS as an additional layer within current 3D CNNs, which use static temporal feature resolutions, we can convert them into much more efﬁcient 3D CNNs with adaptive temporal feature resolutions (ATFR) .”