Inception Transformer

By Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, Shuicheng Yan. Neural Information Processing Systems, 2022

300 citations recorded by the source corpus.

Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information. To tackle this issue, we present a novel and general-purpose Inception Transformer, or iFormer for short, that effectively learns comprehensive features with both high- and low-frequency information in visual data. Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to Transformers. Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high- and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range. Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e. gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high- and low-frequency components across different layers. We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation. For example, our iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-1K, much higher than DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%) with only 1/4 parameters and 1/3 FLOPs. Code and models will be released at https://github.com/sail-sg/iFormer.

View Paper: https://www.semanticscholar.org/paper/dbf6e95cb618f207f029276a6df11f4a9a6313d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Wenhai Wang
2. Zhe Chen
3. Yu Qiao
4. Yanyu Li
5. Junjun He
6. Tong Lu
7. Jifeng Dai
8. Limin Wang

Citations in the corpus

• MambaOut: Do We Really Need Mamba for Vision?* (2024, unknown): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
• Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “…https://github.com/snap-research/EfficientFormer. refine the model architecture and improve training strategies, leading to superior results on most computer vision benchmarks, such as classification [49, 51, 8, 54], segmentation [80, 14, 6], detection [7, 44, 66], and image synthesis [21, 26].”
• Vision Transformer Adapter for Dense Predictions (2022, background): “Recent works (Park & Kim, 2022; Si et al., 2022) show that ViT presents the characteristics of learning low-frequency global signals, while CNN tends to extract high-frequency information ( e.g. , local edges and textures).”