UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning

By Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, Yu Qiao. International Conference on Learning Representations, 2022

368 citations recorded by the source corpus.

It is a challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due to large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. Although 3D convolution can efficiently aggregate local context to suppress local redundancy from a small 3D neighborhood, it lacks the capability to capture global dependency because of the limited receptive field. Alternatively, vision transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation on reducing local redundancy with blind similarity comparison among all the tokens in each layer. Based on these observations, we propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computation and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, our UniFormer achieves 82.9%/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods. For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy respectively. Code is available at https://github.com/Sense-X/UniFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c8831d0629f0eaf7f723317d71bbd60b8eb3c39f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Jiashi Feng
3. Jun Liu
4. Qiuhong Ke
5. Limin Wang
6. Hossein Rahmani
7. Shuicheng Yan
8. Xinchao Wang

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Dacheng Tao
3. Jiashi Feng
4. Luc Van Gool
5. Jun Liu
6. Ming-Hsuan Yang
7. Shuicheng Yan
8. Xiaogang Wang

Citations in the corpus

• AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including image classification [31, 63, 60], object detection [11, 98, 22], semantic/instance segmentation [86], video understanding [8, 2, 33, 57], point cloud modeling [95, 41], 3D Object Rec”
• BiFormer: Vision Transformer with Bi-Level Routing Attention (2023, methodology): “Specifically, in stage i , we use an overlapped patch embedding in the first stage and a patch merging module [25, 37] in the second to fourth stages to reduce the input spatial resolution while increasing the number of channels, followed by N i consecutive BiFormer blocks to transform the features.”
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We evaluate VideoMAE and UniFormerV2 in InternVideo on Kinetics-400 [27], Kinetics-600 [69], Kinetics-700 [70], Something-in-Something-V1 [68], Something-in-Something-V2 [68], ActivityNet [79], HACS [80], and HMDB51 [81].”
• Vision Transformers for Single Image Dehazing (2022, background): “Besides, some works [27], [28], [29], [30] employ convolution in the early stages to introduce the inductive bias.”
• VideoMamba: State Space Model for Efficient Video Understanding (2024, neutral): “In contrast, our VideoMamba introduces a linear-complexity operator for e ffi cient long-term modeling, outperforming existing methods with faster speed and lower GPU consumption.”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Inspired by the success of Transformers in Natural Language Processing (NLP), plenty of works [160], [167] employed Transformers for HAR from videos.”
• Inception Transformer (2022, methodology): “For example, our iFormer-B surpasses UniFormer-B [22], Swin-S [5] by 0 .”
• MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 57, 70], MLP-based [66, 67, 27, 6, 41], convolution-based [15, 19, 79, 56, 77], hybrid [61, 12, 37, 70] or other types of [64, 20] token mixers.”