Are Labels Necessary for Neural Architecture Search?

By Chenxi Liu, Piotr Doll'ar, Kaiming He, Ross B. Girshick, Alan L. Yuille, Saining Xie. 2020

80 citations recorded by the source corpus.

Existing neural network architectures in computer vision -- whether designed by humans or by machines -- were typically found using both images and their associated labels. In this paper, we ask the question: can we find high-quality neural architectures using only images, but no human-annotated labels? To answer this question, we first define a new setup called Unsupervised Neural Architecture Search (UnNAS). We then conduct two sets of experiments. In sample-based experiments, we train a large number (500) of diverse architectures with either supervised or unsupervised objectives, and find that the architecture rankings produced with and without labels are highly correlated. In search-based experiments, we run a well-established NAS algorithm (DARTS) using various unsupervised objectives, and report that the architectures searched without labels can be competitive to their counterparts searched with labels. Together, these results reveal the potentially surprising finding that labels are not necessary, and the image statistics alone may be sufficient to identify good neural architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Xiaodan Liang
3. Quoc V. Le
4. Trevor Darrell
5. In-So Kweon
6. Mohit Bansal
7. Xinlei Chen
8. Saining Xie

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Alan L. Yuille
3. Dacheng Tao
4. Xiaogang Wang
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Yi Yang
8. Jitendra Malik

In Neural Architecture Search & AutoML

1. Chenxi Liu
2. Zhuang Liu
3. Wei Ao
4. Xiaojun Chang
5. Alan L. Yuille
6. Saining Xie
7. Tao Tang
8. Shen Yan

Citations in the corpus

• BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “Without access to any human-annotated labels, unsupervised NAS methods (optimized with pretext tasks [41] or random labels [86]) have been proven capable of achieving comparable performance to supervised NAS methods.”
• ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders (2023, methodology): “In recent years, some efforts have been made to perform architecture search using self-supervised pre-text tasks such as rotation prediction and colorization, as in the case of UnNAS [50].”
• Deep Nets: What have They Ever Done for Vision? (2018, background): “1, there is also recent evidence (Liu et al. 2020) that this process is insensitive to supervision.”
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, background): “There is concurrent work [47] showing that architectures searched without using labels are competitive to their counterparts searched with labels.”
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, unknown): “Several recent works have shown that exploring architectures with search [33] or pruning [8] is possible without supervision, and we plan to evaluate the ability of our method to guide model explorations.”