MoViNets: Mobile Video Networks for Efficient Video Recognition
By D. Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew A. Brown, Boqing Gong. 2021
299 citations recorded by the source corpus.
We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference. 3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets and do not support online inference, making them difficult to work on mobile devices. We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs. First, we design a video network search space and employ neural architecture search to generate efficient and diverse 3D CNN architectures. Second, we introduce the Stream Buffer technique that decouples memory from video clip duration, allowing 3D CNNs to embed arbitrary-length streaming video sequences for both training and inference with a small constant memory footprint. Third, we propose a simple ensembling technique to improve accuracy further without sacrificing efficiency. These three progressive techniques allow MoViNets to achieve state-of-the-art accuracy and efficiency on the Kinetics, Moments in Time, and Charades video action recognition datasets. For instance, MoViNet-A5-Stream achieves the same accuracy as X3D-XL on Kinetics 600 while requiring 80% fewer FLOPs and 65% less memory. Code is available at https://github.com/google-research/movinet.
Subfields: Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Cordelia Schmid
- Anurag Arnab
- Chen Sun
- Luc Van Gool
- Yi Yang
- Xiaogang Wang
- Andrew Zisserman
- Jitendra Malik
In Endocrinology, Diabetes and Metabolism
In Efficiency & Compression
Citations in the corpus
- STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, methodology): “For the tasks with query actions ( e . g ., feasibility/sequence) in questions only, we adopt a pretrained action recognizer MoViNets [19] to recognize seen actions in the situation video as preconditions.”
- Multiview Transformers for Video Recognition (2022, methodology): “Our MTV-L model significantly improves over the previous state-of-the-art [35] by 1.”
- VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, unknown): “5 [36] TimeSformer [6] V ImageNet-21K [54] 62.”