UniFormer: Unifying Convolution and Self-Attention for Visual Recognition

By Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, Yu Qiao. 2022

610 citations recorded by the source corpus.

It is a challenging task to learn discriminative representation from images and videos, due to large local redundancy and complex global dependency in these visual data. Convolution neural networks (CNNs) and vision transformers (ViTs) have been two dominant frameworks in the past few years. Though CNNs can efficiently decrease local redundancy by convolution within a small neighborhood, the limited receptive field makes it hard to capture global dependency. Alternatively, ViTs can effectively capture long-range dependency via self-attention, while blind similarity comparisons among all the tokens lead to high redundancy. To resolve these problems, we propose a novel Unified transFormer (UniFormer), which can seamlessly integrate the merits of convolution and self-attention in a concise transformer format. Different from the typical transformer blocks, the relation aggregators in our UniFormer block are equipped with local and global token affinity respectively in shallow and deep layers, allowing tackling both redundancy and dependency for efficient and effective representation learning. Finally, we flexibly stack our blocks into a new powerful backbone, and adopt it for various vision tasks from image to video domain, from classification to dense prediction. Without any extra training data, our UniFormer achieves <italic>86.3</italic> top-1 accuracy on ImageNet-1 K classification task. With only ImageNet-1 K pre-training, it can simply achieve state-of-the-art performance in a broad range of downstream tasks. It obtains <italic>82.9/84.8</italic> top-1 accuracy on Kinetics-400/600, <italic>60.9/71.2</italic> top-1 accuracy on Something-Something V1/V2 video classification tasks, <italic>53.8</italic> box AP and <italic>46.4</italic> mask AP on COCO object detection task, <italic>50.8</italic> mIoU on ADE20 K semantic segmentation task, and <italic>77.4</italic> AP on COCO pose estimation task. Moreover, we build an efficient UniFormer with a concise hourglass design of token shrinking and recovering, which achieves <bold>2-4</bold><inline-formula><tex-math notation="LaTeX">$\bm {\times }$</tex-math><alternatives><mml:math><mml:mo>×</mml:mo></mml:math><inline-graphic xlink:href="li-ieq1-3282631.gif"/></alternatives></inline-formula> higher throughput than the recent lightweight models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Hongsheng Li
3. Peng Gao
4. Renrui Zhang
5. Bin Xiao
6. Lu Yuan
7. Jingdong Wang
8. Ping Luo

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yu Qiao
3. Dacheng Tao
4. Xiaogang Wang
5. Hongsheng Li
6. Ming-Hsuan Yang
7. Chen Change Loy
8. Yi Yang

In Model Architectures

1. Lu Yuan
2. Bin Xiao
3. Hongsheng Li
4. Ping Luo
5. Jingdong Wang
6. Noel C. F. Codella
7. Yu Qiao
8. Peng Gao

Citations in the corpus

• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “Remove Objects from Image [10, 35] Replace Objects from Image [10, 35] Change Image by the Text [35] Image Question Answering [23] Image-to-Text [23] Text-to-Image [35] Image-to-Edge [45] Edge-to-Image [53] Image-to-Line [16] Line-to-Image [53] Image-to-Hed [44] Hed-to-Image [53] Image-to-Seg [24] S”
• Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “A series of Transformer-based methods [13,27,56,51,36,18,12,6,57,60,25,42] have been developed for high-level vision tasks, including image classification [36,13,27,44,49], object detection [34,48,36,4,6], segmentation [55,51,16,2], etc.”
• DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
• DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, neutral): “ViT [22]/DeiT [73] × APE × X × × × PVT [77] × × APE × X × X × Swin [54] × × RPE X × × X × Twins [13] × × CPE X X × X × GG [94] × × RPE/APE × × X X × Shuffle [38] X × RPE X × X X × MaxViT [75] X X CPE X × X X × CrossFormer [78] X X RPE X × X X X ViL [104] × × RPE/APE X X × X × NAT [32] X X RPE X × × ”
• ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “As a key component, the model architectures adopted by existing video methods has expanded from CNNs [32, 68, 19, 18, 77, 69, 72, 48, 41, 45] to Transformers [17, 40, 38, 44, 2, 6].”
• The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot (2023, background): “Unlike CNNs, which rely on convolutional operations, ViT employs self-attention mechanisms that allow it to model long-range dependencies and global context within images [28, 29].”
• CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, background): “SegFormer [61], PVT [62], Swin [24], and UniFormer [63] create hierarchical structures to make use of multi-resolution features.”
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, methodology): “Motivated by this, Vision Transformers [14] and DETR [6] introduce the transformer architecture into computer vision, and stimulate follow-up works to effectively apply transformers to a wide range of vision tasks, such as image classification [43, 30, 31], object detection [66, 64, 16], semantic se”
• Expanding Language-Image Pretrained Models for General Video Recognition (2022, neutral): “1 3 × 3 455 7 Methods with ImageNet pretraining Uniformer-B [25] IN-1k 32 83.”