Best Practices for Scientific Research on Neural Architecture Search

By Marius Thomas Lindauer, Frank Hutter. 2019

155 citations recorded by the source corpus.

Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empirical evaluations in the young field of NAS is still lacking behind that of other areas of machine learning. To address this issue, we describe a set of possible issues and ways to avoid them, leading to the NAS best practices checklist available at this http URL

Subfields: Artificial Intelligence, Materials Chemistry, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Neural Architecture Search & AutoML

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “However, the community still lacks the variety of benchmarks that allow fully reproducible NAS experiments, as Lindauer and Hutter pointed out [27].”
NAS evaluation is frustratingly hard (2019, methodology): “Comparison between published search algorithms for NAS is therefore either very difficult (complex training protocols with no code available) or simply impossible (different search spaces), as previously pointed out (Li & Talwalkar, 2019; Sciuto et al., 2019; Lindauer & Hutter, 2019).”
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Since the DARTS search space is not a tabular dataset, we cannot fairly compare to other methods which use substantially different training and testing pipelines (Lindauer and Hutter 2019).”
CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability (2021, background): “…conclusions have been reached in related works focused on specifics of reinforcement learning (Li & Talwalkar, 2019), neural architecture search (Lindauer & Hutter, 2020), human-centered machine learning model cards (Mitchell et al., 2019), or general dataset sheet specifications (Bender &…”
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? (2020, methodology): “We follow the NAS best practices checklist [22] to conduct our experiments.”
A Study on Encodings for Neural Architecture Search (2020, methodology): “Recent papers have highlighted the need for fair and reproducible NAS comparisons [8, 24, 9].”
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search (2020, background): “This is a result of several factors, such as unavailable original implementations, differences in the employed search spaces, training or evaluation pipelines, hyperparameter settings, and even pseudorandom number seeds (Lindauer & Hutter, 2019).”
Local Search is State of the Art for NAS Benchmarks (2020, methodology): “To promote reproducible research, we discuss how our experiments follow the best practices checklist (Lindauer and Hutter, 2019) in Appendix B.”
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size (2020, background): “It is essentially not clear if the reported improvements have come from hyper-parameter settings, re-training pipelines, random seeds, or the improvements of the searching algorithm itself [25].”
Neural Architecture Search with Reinforce and Masked Attention Autoregressive Density Estimators (2020, methodology): “In order for better evaluation and reproduction of our research we address all the items in the checklist as mentioned in (Lindauer & Hutter, 2019).”
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, unknown): “This practice impedes assertions about the statistical signiﬁcance of the reported results, recently brought into focus by several authors (Yang et al., 2019; Lindauer & Hutter, 2019; Shu et al., 2020; Yu et al., 2020).”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “Although it is common practice to optimize architecture and HPCs in sequence, recent evidence suggests that they should be jointly optimized (Lindauer & Hutter, 2020; Zela et al., 2018).”
Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, methodology): “To promote reproducible research, we discuss how our experiments follow the best practices checklist [19] in Appendix B, and we release our code at https://github.”