Escaping the Big Data Paradigm with Compact Transformers

By Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, Humphrey Shi. 2021

585 citations recorded by the source corpus.

With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

SOCRATES: Introducing Depth in Visual Wildlife Monitoring Using Stereo Vision (2022, background): “This is motivated by the observation that vision transformers require more training data to perform well, compared to CNNs [43,44].”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Besides, LeViT, CoAT, and CCT contain CNN structures to obtain inductive biases, which still suffer from catastrophic forgetting in online CL.”
Transformers in Vision: A Survey (2021, methodology): “Compact Convolutional Transformer (CCT) [77] introduces a new sequence pooling scheme, and incorporates convolutional blocks (conv-poolreshape) for tokenization.”
Auto-scaling Vision Transformers without Training (2022, background): “Although recent works attempt to enhance ViT’s data and resource efﬁciency (Touvron et al., 2020; Hassani et al., 2021; Pan et al., 2021; Chen et al., 2021d), the heavy computation cost (e.g., quadratic with respect to the number of tokens) is still overwhelming, compared with training CNNs.…”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, background): “Although promising, this increases the supervised training complexity, and existing reported performance on data efficient benchmarks (Hassani et al. 2021; Chen et al. 2021) still significantly underperforms convolutional networks.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Patch embedding is often implemented with a non-overlapping convolution layer that has large kernel size and stride [3, 55].”
Neighborhood Attention Transformer (2022, methodology): “Some works focused on data efficiency, with minor changes in the architecture [26, 33, 11, 27, 10], while others focused on efficiency and transferrability to downstream tasks [18, 34, 28, 10].”
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer (2021, methodology): “CvT [40] and CCT [15] introducing convolutions into the Vision Transformer architecture to merge the beneﬁts of Transformers with the beneﬁts of CNNs for image recognition tasks.”
Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “Similar to the convolutional stem in Vision Transformer (Xiao et al., 2021b; Hassani et al., 2021), we apply a convolution layer in each SPS block to introduce inductive bias into Spikformer.”
Aggregating Nested Transformers (2021, methodology): “The CIFAR results of compared transformer based methods, besides CCT-7/ 3 × 1 [22], in Table 1 are trained by us.”
Continual Learning with Lifelong Vision Transformer (2022, methodology): “Unlike vanilla self-attention in vision transformers [21, 25, 32, 74] that derives the attention map by computing similarity between self-queries and self-keys, we propose intertask attention mechanism to obtain attention map by computing the affinities between self-queries and a learnable external ”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “For reducing the computation complexity of the vision transformer, many works propose new modules and refine architecture design [10,20,26,37,41,61], while others eliminate redundancies in attention mechanism [9, 14, 29, 44, 60, 71, 74].”
Multi-modal Deep Learning (2024, methodology): “Employing the generated feature map, it is subsequently fed into the ImageNet Pretrained Compact Convolutional Transformer (CCT) [20], which yields a 7,200-dimensional feature vector.”