GLiT: Neural Architecture Search for Global and Local Image Transformer

By Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, Wanli Ouyang. 2021

104 citations recorded by the source corpus.

We introduce the first Neural Architecture Search (NAS) method to find a better transformer architecture for image recognition. Recently, transformers without CNN-based backbones are found to achieve impressive performance for image recognition. However, the transformer is designed for NLP tasks and thus could be sub-optimal when directly used for image recognition. In order to improve the visual representation ability for transformers, we propose a new search space and searching algorithm. Specifically, we introduce a locality module that models the local correlations in images explicitly with fewer computational cost. With the locality module, our search space is defined to let the search algorithm freely trade off between global and local information as well as optimizing the low-level design choice in each module. To tackle the problem caused by huge search space, a hierarchical neural architecture search method is proposed to search the optimal vision transformer from two levels separately with the evolutionary algorithm. Extensive experiments on the ImageNet dataset demonstrate that our method can find more discriminative and efficient trans-former variants than the ResNet family (e.g., ResNet101) and the baseline ViT for image classification. The source codes are available at https://github.com/bychen515/GLiT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “Another recent efort studies the trade-of between global and local information in Transformers in the context of vision applications [27].”
TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, methodology): “Besides, TinyViT surpasses automatically searched models, such as AutoFormer [11] and GLiT [9].”
Fast Vision Transformers with HiLo Attention (2022, methodology): “Besides, we are also not directly comparable with NAS-based methods [2, 4] as LITv2 is manually designed.”
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, result): “We also compare our models with recent efficient vision transformers [5, 9, 50, 51, 56] in Tab.”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, background): “Thanks to significantly more parallelization and competitive performance, transformer becomes a prevailing architecture in both language modeling [14,37] and vision community [15,41,6,5].”
Vision Transformer with Deformable Attention (2022, methodology): “This alternate design of attention blocks with local and global receptive ﬁelds helps the model learn strong representations, sharing a similar pattern in GLiT [5], TNT [15] and Point-DAT”
Inception Transformer (2022, background): “GLiT [52] also adopt parallel manner but it directly concatenate the features from convolution and attention branches as the mixer output, lacking the fusion of features in different frequencies.”
A Survey on Vision Transformer (2020, methodology): “, Scaling-ViT [269], ViTAS [205], AutoFormer [28] and GLiT [24].”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, methodology): “GLiT [5] introduces a locality module and use neural architecture search to find an efficient backbone.”