Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS
By Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, Quoc V. Le. 2020
131 citations recorded by the source corpus.
Efficient Neural Architecture Search methods based on weight sharing have shown good promise in democratizing Neural Architecture Search for computer vision models. There is, however, an ongoing debate whether these efficient methods are significantly better than random search. Here we perform a thorough comparison between efficient and random search methods on a family of progressively larger and more challenging search spaces for image classification and detection on ImageNet and COCO. While the efficacies of both methods are problem-dependent, our experiments demonstrate that there are large, realistic tasks where efficient search methods can provide substantial gains over random search. In addition, we propose and evaluate techniques which improve the quality of searched architectures and reduce the need for manual hyper-parameter tuning.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Xiaodan Liang
- Jiahui Yu
- Thomas S. Huang
- Xuanyi Dong
- Mingxing Tan
- Xiaojun Chang
- Hanxiao Liu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Yi Yang
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Quoc V. Le
- Ming-Hsuan Yang
- Alan L. Yuille
Citations in the corpus
- BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models (2020, methodology): “We train our big single-stage model on ImageNet [7] using same settings as [16,27,31,32].”
- Neural Predictor for Neural Architecture Search (2019, methodology): “These reproductions come from the TuNAS [3] codebase.”
- MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “We make use of TuNAS [1], a one-shot NAS framework which uses aggressive weight sharing that is well-suited for computation intensive video models.”
- BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search (2021, methodology): “Our BossNAS out performs recent NAS methods [66, 5] designed particularly for network size search spaces, proving the generalization ability of our method on specified search spaces and relatively small datasets.”
- AutoHAS: Efficient Hyperparameter and Architecture Search. (2020, methodology): “In addition, when the search space includes architecture choices, we also uses the warm-up technique described in Bender et al. (2020). For these meta hyperparameters, we use Adam, momentum as 0.”
- NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, background): “TAS and FBNetV2 optimize the architecture parameters in a differentiable way, and TuNAS uses REINFORCE.”
- MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, unknown): “To effectively instantiate the UIB blocks, we adopt TuNAS [4] with tailored enhancements for improved performance.”
- Searching for Fast Model Families on Datacenter Accelerators (2021, methodology): “We use the same multiobjective reward mechanism as in [56] to ensure fair comparison, although different objective function forms, such as additive forms [10], can potentially produce even better results.”