BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

By Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang, Xiaodan Liang, Xiaojun Chang. 2021

124 citations recorded by the source corpus.

A myriad of recent breakthroughs in hand-crafted neural architectures for visual recognition have highlighted the urgent need to explore hybrid architectures consisting of diversified building blocks. Meanwhile, neural architecture search methods are surging with an expectation to reduce human efforts. However, whether NAS methods can efficiently and effectively handle diversified search spaces with disparate candidates (e.g. CNNs and transformers) is still an open question. In this work, we present Block-wisely Self-supervised Neural Architecture Search (BossNAS), an unsupervised NAS method that addresses the problem of in-accurate architecture rating caused by large weight-sharing space and biased supervision in previous methods. More specifically, we factorize the search space into blocks and utilize a novel self-supervised training scheme, named ensemble bootstrapping, to train each block separately before searching them as a whole towards the population center. Additionally, we present HyTra search space, a fabric-like hybrid CNN-transformer search space with searchable down-sampling positions. On this challenging search space, our searched model, BossNet-T, achieves up to 82.5% accuracy on ImageNet, surpassing EfficientNet by 2.4% with comparable compute time. Moreover, our method achieves superior architecture rating accuracy with 0.78 and 0.76 Spearman correlation on the canonical MBConv search space with ImageNet and on NATS-Bench size search space with CIFAR-100, respectively, surpassing state-of-the-art NAS methods. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “BossNAS [264] searches for a hybrid architecture (CNN and Transformer).”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, background): “While vision transformer suffers from the necessity large-scaled dataset [48], many recent works try to encode strong inductive prior by either combining it with convolutional layer [60, 32, 64, 62] or introducing 2D-hierarchical structure to vision transformer [34, 56, 14, 9].”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, background): “Other works introduce the pyramid/hierarchical structure to transformer [52, 53, 54] or combine it with convolutional layers [55, 56].”
Auto-scaling Vision Transformers without Training (2022, background): “In contrast, BoneNAS (Li et al., 2021) requires 10 GPU days to search a single architecture.”
DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
LocalViT: Analyzing Locality in Vision Transformers (2021, unknown): “There are also some works that introduce hybrid architectures of CNNs and transformers [15], [16].”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “…observe that DARTS is somewhat unreliable (Zela et al., 2020a; Chen & Hsieh, 2020; Li & Talwalkar, 2019; Sciuto et al., 2019; Zhang et al., 2020c; Li et al., 2021a; Zhang et al., 2020b) since it does not consistently yield excellent solutions, performing even worse than random search in some…”
AutoFormer: Searching Transformers for Visual Recognition (2021, background): “BossNAS searches for CNN-transformer hybrids, while ours for pure transformers.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, unknown): “…[145], multi-task learning [157], [158], transfer learning [159], [160], [161], graph NNs [147], [162], [163], reinforcement learning [164], [165], [166], few-shot learning [167], [168], neural architecture search [169], robust learning [146], [170], [171], [172], and meta-learning [173], [174].”