Patches Are All You Need?

By Asher Trockman, J. Kolter. 2022

538 citations recorded by the source corpus.

Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/locuslab/convmixer.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

A ConvNet for the 2020s (2022, background): “A concurrent work ConvMixer [4] demonstrates that, in small-scale settings, depthwise convolution can be used as a promising mixing strategy.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, methodology): “ConvMixer [87] uses up to 9×9 convolutions to replace the “mixer” component of ViTs [34] or MLPs [84, 85].”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “RepMixer Convolutional mixing was first introduced in ConvMixer [54].”
Transolver: A Fast Transformer Solver for PDEs on General Geometries (2024, methodology): “Previous methods attempt to tackle the complexity problem by introducing linear attention (Hao et al., 2023; Tran et al., 2023), but directly applying the attention to massive mesh points may overwhelm the model from learning informative relations (Wu et al., 2022).”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Furthermore, we do not use depth-wise convolution to replace pooling [70] as we focus on building architecture without the aid of lightweight convolutions.”
YOLO-FaceV2: A Scale and Occlusion Aware Face Detector (2022, methodology): “Second, inspired by FAN and ConvMixer [31], we redesign a multi-head attention network to compensate for the loss of occluded face response values.”
Wavelet Convolutions for Large Receptive Fields (2024, methodology): “Trockman and Kolter [53] have attempted to mimic the ViT architecture using only convolutions and have shown impressive results by us-ing 9 × 9 convolutions to replace the attention (or “mixer") component.”
DaViT: Dual Attention Vision Transformers (2022, methodology): “We use a simple triangular learning rate schedule [45] as in [51].”