Conformer: Local Features Coupling Global Representations for Visual Recognition

By Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, Qixiang Ye. IEEE International Conference on Computer Vision, 2021

757 citations recorded by the source corpus.

Within Convolutional Neural Network (CNN), the convolution operations are good at extracting local features but experience difficulty to capture global representations. Within visual transformer, the cascaded self-attention modules can capture long-distance feature dependencies but unfortunately deteriorate local feature details. In this paper, we propose a hybrid network structure, termed Conformer, to take advantage of convolutional operations and self-attention mechanisms for enhanced representation learning. Conformer roots in the Feature Coupling Unit (FCU), which fuses local features and global representations under different resolutions in an interactive fashion. Conformer adopts a concurrent structure so that local features and global representations are retained to the maximum extent. Experiments show that Conformer, under the comparable parameter complexity, outperforms the visual transformer (DeiT-B) by 2.3% on ImageNet. On MSCOCO, it outperforms ResNet-101 by 3.7% and 3.6% mAPs for object detection and instance segmentation, respectively, demonstrating the great potential to be a general backbone network. Code is available at github.com/pengzhiliang/Conformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5faf75b5c5a4d83bd6407b4aba8fb0bccd7fa31d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jiashi Feng
3. Lu Yuan
4. Luc Van Gool
5. Jianfeng Gao
6. Yu Qiao
7. Alan L. Yuille
8. Jing Zhang

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Jiashi Feng
4. Ming-Hsuan Yang
5. Yu Qiao
6. Yi Yang
7. Xiaogang Wang
8. Alan L. Yuille

Citations in the corpus

• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “Conformer [49], the most relevant concurrent work to us, employs a unit to explore inter-block interactions between parallel convolution and transformer blocks.”
• Visformer: The Vision-friendly Transformer (2021, background): “…81.2 6.7 44.2 PVTv2-B2-Li [58] 82.1 3.9 22.6 PVTv2-B2 [58] 82.0 4.0 25.4 Swin-T [42] 81.3 4.5 29 CvT-13 [32] 81.6 4.5 20 CvT-13-NAS [32] 82.2 4.1 18 CvT-13(384) [32] 83.0 16.3 20 Conformer-Ti [31] 81.3 5.2 23.5 T2T-ViT t -14 [52] 80.7 5.2 21.5 T2T-ViT t -19 [52] 81.4 8.4 39.0 BoTNet-S1-59 [29] 81.”
• Dual Aggregation Transformer for Image Super-Resolution (2023, background): “Firstly , considering the mis-alignment between local (DW-Conv) and global (SA) features [18, 24], the two branches cannot be fused effectively.”
• HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling (2022, background): “Convolution + Transformers Conformer[35] CNN branch 83.”
• DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “, image classification [21], [46], [47], [54], [61], [73], object detection [5], [17], [21], [24], [32], [44], [58], [104],”
• HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classification (2022, methodology): “The models, such as VitAE, StoHis, TransFuse, CMT, Conformer and so on,Xu et al. [2021], Fu et al. [2022], Zhang et al. [2021], Guo et al. [2022a], Peng et al. [2021] not only inherit the structural advantages of CNN and Transformer, but also verify that the coupling of local features and global…”
• Mobile-Former: Bridging MobileNet and Transformer (2021, methodology): “Parallel architectures: compared to parallel architectures like Conformer [27] and CrossViT [2], our Mobile-Former uses significantly fewer tokens (6 vs.”
• On the Integration of Self-Attention and Convolution (2021, methodology): “, SASA [37], LRNet [20], AA-ResNet [3], BoTNet [40], T2T-ViT [51], ConViT [12], CVT [47], ConT [50] and Conformer [35].”
• Vision Transformer Adapter for Dense Predictions (2022, background): “Conformer [59] proposed the first dual network to combine CNN with transformer.”
• Refiner: Refining Self-attention for Vision Transformers (2021, methodology): “[32] replaces the feed forward block in the transformer block with an inverted residual one [41, 26] and [37] propose to build two concurrent branches: one with convolution and the other one with transformer.”
• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, methodology): “Recently, some works try to introduce the intrinsic IB of CNNs into vision transformers explicitly (Han et al., 2021; Peng et al., 2021; Graham et al., 2021; Li et al., 2021; d’Ascoli et al., 2021; Yan et al., 2021; Wu et al., 2021; Yuan et al., 2021a; Chen et al., 2021b; Liu et al., 2021).”
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, methodology): “Conformer (Peng et al., 2021) Image recognition & object detection, Classification 2021 Encoder Not mentioned in paper N/A LibriSpeech”