Focal Modulation Networks

By Jianwei Yang, Jianfeng Gao. 2022

462 citations recorded by the source corpus.

We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation mechanism for modeling token interactions in vision. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (ii) gated aggregation to selectively gather contexts for each query token based on its content, and (iii) element-wise modulation or affine transformation to inject the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational costs on the tasks of image classification, object detection, and segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K in 224 resolution, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 and 384, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1\times outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3\times schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and Mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. Using huge FocalNet and DINO, we achieved 64.3 and 64.4 mAP on COCO minival and test-dev, respectively, establishing new SoTA on top of much larger attention-based models like Swinv2-G and BEIT-3. Code and checkpoints are available at https://github.com/microsoft/FocalNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “We also use Focal-T [48] in our ablation studies following [60].”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Most existing efforts devote to model-centric methods such as efficient network architectures [75], smarter training objectives [76], and scaling up model size [22, 84].”
Rewrite the Stars (2024, unknown): “GLU [48], etc. ), Computer Vision ( i.e. , FocalNet [60], Hor-Net [45], VAN [18], etc. ), and more.”
DETRs Beat YOLOs on Real-time Object Detection (2023, methodology): “Existing large DETR models [3, 6, 30, 38, 41, 43] have demonstrated impressive performance on COCO test-dev [19] leaderboard.”
Visual attention network (2022, methodology): “The visualization method follows FocalNet [105], which visualizes the absolute value of attention map.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “VAN [19] and FocalNet [61] use gated convolutions to perform input-adaptive attention and adopts large-kernel dilated convolutions and multiple successive 3×3 convolutions respectively to produce the weights.”
DETRs with Collaborative Hybrid Assignments Training (2022, unknown): “DINO [33] FocalNet-H [32] 746M IN-22K (14M) Objects365 64.”
MambaOut: Do We Really Need Mamba for Vision?* (2024, unknown): “For vision tasks, as image classification on Ima-*”
InceptionNeXt: When Inception Meets ConvNeXt (2023, unknown): “ConvNeXt is not alone: More and more works have shown similar observations [14, 18, 24, 38, 51, 68, 76, 78], like RepLKNet [14] and SLaK [38].”
MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 57, 70], MLP-based [66, 67, 27, 6, 41], convolution-based [15, 19, 79, 56, 77], hybrid [61, 12, 37, 70] or other types of [64, 20] token mixers.”
Segment Everything Everywhere All at Once (2023, methodology): “For the vision backbone, we use FocalT [58], DaViT-d3 (B), and DaViT-d5 (L) [59].”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, background): “Thanks to the strong universal representation learned by Florence-2 pre-training, we do not require longer training epochs, such as 36 epochs in [51, 81, 85, 86], or 100 epochs in [46], to achieve better results.”