Towards modular and programmable architecture search

By Renato Negrinho, Darshan Patil, N. Le, Daniel C. Ferreira, Matthew R. Gormley, Geoffrey J. Gordon. 2019

31 citations recorded by the source corpus.

Neural architecture search methods are able to find high performance deep learning architectures with minimal effort from an expert. However, current systems focus on specific use-cases (e.g. convolutional image classifiers and recurrent language models), making them unsuitable for general use-cases that an expert might wish to write. Hyperparameter optimization systems are general-purpose but lack the constructs needed for easy application to architecture search. In this work, we propose a formal language for encoding search spaces over general computational graphs. The language constructs allow us to write modular, composable, and reusable search space encodings and to reason about search space design. We use our language to encode search spaces from the architecture search literature. The language allows us to decouple the implementations of the search space and the search algorithm, allowing us to expose search spaces to search algorithms through a consistent interface. Our experiments show the ease with which we can experiment with different combinations of search spaces and search algorithms without having to implement each combination from scratch. We release an implementation of our language with this paper.

Subfields: Artificial Intelligence, Information Systems, Materials Chemistry, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Chen Liang
3. David R. So
4. Esteban Real
5. Corey Nolet
6. Joshua Patterson
7. Sebastian Raschka
8. Frank Hutter

In Information Systems

1. Chen Liang
2. Sebastian Raschka
3. Quoc V. Le

In Materials Chemistry

1. Chen Liang
2. Quoc V. Le
3. David R. So

In Neural Architecture Search & AutoML

1. Quoc V. Le
2. Chen Liang
3. Corey Nolet
4. Joshua Patterson
5. Sebastian Raschka
6. Esteban Real
7. David R. So
8. Frank Hutter

Citations in the corpus

• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Second, constrained search spaces need to be carefully composed (Zoph et al., 2018; So et al., 2019; Negrinho et al., 2019), thus creating a new burden on researchers and undermining the purported objective of saving their time.”
• Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020, background): “e DNN architectures from a predefined set of low-level building blocks. This type of model generation is referred to as neural architecture search (NAS) [64], which is a subfield of architecture search [65,66]. The overarching theme in the development of architecture search algorithms is to define a se”