Global Filter Networks for Image Classification

By Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, Jie Zhou. 2021

726 citations recorded by the source corpus.

Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “This also aligns with a recent trend on using Transformers in computer vision [6, 8] and treating Fourier transform as a lightweight replacement to the selfattention [24, 35].”
A ConvNet for the 2020s (2022, methodology): “GFNet [56] adopts Fast Fourier Transform (FFT) for token mixing.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “Besides, both [38] and [72] do not evaluate their models on strong baselines, e.”
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers (2021, methodology): “Note that the concurrent GFN work Rao et al. (2021) is a special case of FNO, when W m,n is diagonal and the channels are separable.”
Filter-enhanced MLP is All You Need for Sequential Recommendation (2022, methodology): “More recently, GFNet [39] adopts 2D Fourier transformwith the global filter layer to learn long-term spatial dependencies in the frequency domain, which shows competitive performance to Transformer-based models.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “The GF layer [44] multiplies the frequency domain features with learnable global filters, which is equivalent to a convolution in the spatial domain with a global kernel size and circular padding.”
One Fits All: Power General Time Series Analysis by Pretrained LM (2023, methodology): “…by its great success in natural language processing and computer vision Vaswani et al. (2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), transformer has been introduced to various time series tasks with promising results Wen et al. (2023), especially for time series…”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, background): “) in a supervised learning [12, 18, 29, 34] or self-supervised learning manner [3, 8, 15, 16], and then finetunes the models on various downstream tasks.”
MetaFormer is Actually What You Need for Vision (2021, background): “Some recent approaches [31, 38, 39, 44] explore other types of token mixers within the MetaFormer architecture, and have demonstrated encouraging performance.”
Fast Vision Transformers with HiLo Attention (2022, methodology): “In Figure 5, we visualize the magnitude of frequency component [42] by applying Fast Fourier Transform (FFT) to the output feature maps from Hi-Fi and Lo-Fi attentions, respectively.”
Wavelet Convolutions for Large Receptive Fields (2024, methodology): “In this experiment, we use MobilenetV2 [48] where each depth-wise convolution is replaced with RepLK [11], GFNet [45], FFC [4], and the proposed WTConv layer.”
MAXIM: Multi-Axis MLP for Image Processing (2022, methodology): “Very recent techniques such as FNet [43] and GFNet [74] demonstrate the simple Fourier Transform can be used as a competitive alternative to either self-attention or MLPs.”
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting (2022, background): “…success of Transformer Vaswani et al. (2017) in the NLP and CV communities Vaswani et al. (2017); Devlin et al. (2019); Dosovitskiy et al. (2021); Rao et al. (2021), it also shows promising performance in capturing long-term dependencies Zhou et al. (2021); Wu et al. (2021); Zhou et al. (2022);…”
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification (2021, background): “Recent progress has demonstrated the variants of transformers can also be a competitive alternative to CNNs and achieve promising results on different vision tasks including image classification [8, 25, 20, 35, 23], object detection [2], semantic segmentation [34, 5] and 3D analysis [31, 33].”
FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022, methodology): “Following the recent success in NLP and CV community (Vaswani et al., 2017; Devlin et al., 2019; Dosovitskiy et al., 2021; Rao et al., 2021), Transformer (Vaswani et al.”
MetaFormer Baselines for Vision (2022, methodology): “Nevertheless, some work [66, 36, 85, 22, 56] found that, by replacing the attention module in Transformers with simple operators like spatial MLP [66, 67, 65] or Fourier transform [36], the resultant models still produce encouraging performance.”
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, methodology): “Another representative work was Global Filter Network (GFNet) [63], which optimized the spatial connection weights in the Fourier domain.”