Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

By Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, Mi Zhang. 2020

126 citations recorded by the source corpus.

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency. To explain these observations, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.

Subfields: Artificial Intelligence, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Colin White
2. Yash Savani
3. Willie Neiswanger
4. Sam Nolen

In Materials Chemistry

1. Yash Savani
2. Colin White
3. Willie Neiswanger

Citations in the corpus

• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “We use a common evaluation pipeline which is to train for 600 epochs with cutout and auxiliary tower (Liu, Simonyan, and Yang 2018; Li and Talwalkar 2019; Yan et al. 2020), where the state of the art is around 2.”
• Multi-View Encoders for Performance Prediction in LLM-Based Agentic Workflows (2025, unknown): “The resulting representations improve sample efficiency for downstream prediction, in line with observations in NAS (White et al., 2020; Yan et al., 2020; 2021; Akhauri & Abdelfattah, 2024; Trirat & Lee, 2025b).”