FNet: Mixing Tokens with Fourier Transforms

By James Lee-Thorp, J. Ainslie, Ilya Eckstein, Santiago Ontañón. 2021

751 citations recorded by the source corpus.

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that “mix” input tokens. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the “efficient Transformers” on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models outperform Transformer counterparts.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

In Signal Processing

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, neutral): “, 2019), alternative non Transformer-based architectures (Tolstikhin et al., 2021; Tay et al., 2021a; 2020a; Lee-Thorp et al., 2021).”
Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “This also aligns with a recent trend on using Transformers in computer vision [6, 8] and treating Fourier transform as a lightweight replacement to the self-attention [24, 35].”
The Efficiency Misnomer (2021, background): “That said, it is still common practice to parametermatch models to make ‘fair’ comparisons (Mehta et al., 2020; Lee-Thorp et al., 2021; Tay et al., 2020a; Xue et al., 2021; Wightman et al., 2021), even if one model is in reality slower or faster than another.”
Efficient Transformers: A Survey (2020, methodology): “Different mixing operations have been proposed, such as the G-MLP [47], FNet [44].”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “Similarly, Lee-Thorp et al. [41] achieve faster training by substituting self-attention with unparametrized Fourier Transform.”
Blending Anti-Aliasing into Vision Transformer (2021, background): “A recent work FNet [41], which replaces self-attention in BERT [42] with Fourier Transforms, may also share the same spirit.”
Large-Scale Differentially Private BERT (2021, background): “, 2020), more efficient architectures such as MLP-Mixer, and F-NETs (Tolstikhin et al., 2021; Lee-Thorp et al., 2021) for longer sequences, multistep training at mega-batch sizes (Choi et al.”
A Tutorial on Fluid Antenna System for 6G Networks: Encompassing Communication Theory, Optimization Methods and Hardware Designs (2024, positive): “Additionally, there exist spatial and frequency domain interaction mechanisms, such as the spatial MLP (e.g., DynaMixer [130]) and the dynamic fast Fourier transform (FFT) (e.g., FNet [131]), as shown in Fig.”
Global Filter Networks for Image Classification (2021, methodology): “We show that GFNet outperforms the ResMLP [39], FNet [23] and models with local depthwise convolutions.”
ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, background): “(Lee-Thorp et al., 2021) replaces self-attention with a nonlearnable Fourier Transform and verifies it to be an effective mixing mechanism.”
MetaFormer is Actually What You Need for Vision (2021, background): “For example, [31] replaces attention with Fourier Transform and still achieves around 97% of the accuracy of vanilla transformers.”
FITS: Modeling Time Series with 10k Parameters (2023, background): “Another approach, FNet (Lee-Thorp et al., 2022), leverages frequency domain multiplication to replace convolution operations, thereby reducing computational overhead.”
MAXIM: Multi-Axis MLP for Image Processing (2022, methodology): “Very recent techniques such as FNet [43] and GFNet [74] demonstrate the simple Fourier Transform can be used as a competitive alternative to either self-attention or MLPs.”
Factorized Fourier Neural Operators (2021, methodology): “Two notable examples of deep learning models that employ Fourier representations are FNet (Lee-Thorp et al., 2021) for encoding semantic relationships in text classification and FNO (Li et al.”
FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022, background): “(Lee-Thorp et al., 2021) find that replacing the self-attention layer in a transformer encoder with a standard Fourier Transform achieves much shorter training time with small accuracy loss on the GLUE benchmark.”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “Liquid S4 used the original (13)This idea is also leveraged in FNet (Lee-Thorp et al., 2021), where the attention mechanism is replaced with a simpler linear token-mixing strategy.”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Recent works have been focused on improving the architectures of attention models, including sparse attention [33, 34, 35, 36, 37, 38, 39], enlarged receptive fields [40, 41], pyramidal designs [42, 43, 44], efficient alternatives [45, 46, 47], etc.”
Simplified State Space Layers for Sequence Modeling (2022, background): “[33] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon.”
MetaFormer Baselines for Vision (2022, background): “However, it is shown in MLP-Mixer [66] and FNet [36] that, by replacing attention in Transformer with spatial MLP [65] and Fourier transform, the resulting models still deliver competitive results.”