Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution
By Thomas Elsken, Jan Hendrik Metzen, Frank Hutter. 2018
546 citations recorded by the source corpus.
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1)the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption, (2) most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.
Subfields: Artificial Intelligence, Neural Architecture Search & AutoML, Evolutionary Computation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Dacheng Tao
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Trevor Darrell
In Neural Architecture Search & AutoML
In Evolutionary Computation
Citations in the corpus
- Primer: Searching for Efficient Transformers for Language Modeling (2021, background): “This approach, which we call an implicit efficiency objective by fixed training budget, contrasts previous architecture search works that explicitly aim to reduce training or inference step time when optimizing for efficiency [26, 27, 28, 29].”
- Squeeze-and-Excitation Networks (2017, methodology): “fc, [32, 512] × 4 14× 14 conv, 1× 1, 256 conv, 3× 3, 256 conv, 1× 1, 1024 × 6 conv, 1× 1, 256 conv, 3× 3, 256 conv, 1× 1, 1024”
- Advances and Open Problems in Federated Learning (2019, background): “He, M. Annavaram, and S. Avestimehr, “FedNAS: Federated deep learning via neural architecture search,” arXiv preprint arXiv:2004.08546, 2020.”
- MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (2020, methodology): “The third method is gradient-based [17, 28, 5, 19, 4], which relaxes the categorical design choices to continuous variables and then leverages the efficient gradient backpropagation so that it can finish searching within as little as several GPU days.”
- A Comprehensive Survey of Neural Architecture Search (2020, background): “…also quickly applied in the fields of object detection [65, 75, 111, 118], semantic segmentation [63, 64, 120], adversarial learning [53], architectural scaling [114, 122, 124], multi-objective optimization [39, 115, 125], platform-aware [28, 34, 103, 117], data augmentation [121, 123] and so on.”
- Fast Neural Network Adaptation via Parameter Remapping and Architecture Search (2020, background): “Moreover, some NAS works (Pham et al., 2018; Fang et al., 2019a; Elsken et al., 2019) apply parameters sharing on child models to accelerate the search process.”
- Understanding and Robustifying Differentiable Architecture Search (2019, background): “…methods are often very expensive, various work focuses on reducing the search costs by, e.g., employing network morphisms [Cai et al., 2018a,b, Elsken et al., 2017, 2019b], weight sharing within one-shot models [Saxena and Verbeek, 2016, Bender et al., 2018, Pham et al., 2018] or…”
- Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, background): “Exemplar works include hyper-networks [40], network morphism [6] and shared weights [30].”
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search (2019, methodology): “The first type of architecture search methods (Liu et al., 2018b; Xie & Yuille, 2017; Real et al., 2017; Elsken et al., 2019; Real et al., 2019; Miikkulainen et al., 2019) adopted evolutionary algorithms, which assumed the possibility of applying genetic operations to force a single architecture or ”
- Continual and Multi-Task Architecture Search (2019, background): “Elsken et al. (2019) and So et al. (2019) proposed evolutionary architecture search algorithms that dynamically allocate more resources for promising architecture candidates, but these works are different from us in that they do not consider the case where we have continual incoming-data from…”
- Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “Some recent approaches consider using the multi-objective evolutionary architecture search to optimize training time, complexity and performance [76, 36] of the network.”
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization (2019, background): “, 2018)1 and neural architecture search (NAS) (Baker et al., 2017; Zoph and Le, 2017; Real et al., 2017; Elsken et al., 2019; Liu et al., 2019) that tries to automate this process by casting it as an optimization problem.”
- A Survey on Neural Architecture Search (2019, neutral): “Elsken et al. (2019) Elsken et al. (2019) propose another evolutionary algorithm which shares many common aspects with their earlier work (Elsken et al., 2018) (see Section 3.2).”
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, background): “This proliferation of choices has fueled research into neural architecture search (NAS) , which casts the discovery of new architectures as an optimization problem (Baker et al., 2017; Zoph & Le, 2016; Real et al., 2017; Elsken et al., 2018).”
- NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search (2018, methodology): “Elsken et al. present the LEMONADE method in Elsken et al. (2018) which is formulated to develop networks with low error and number of parameters through custom designed approximate network morphisms Wei et al. (2016b).”
- Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “Neural Architecture Search (NAS): Many recent works search the space of deep learning architectures to find ones that perform well (Zoph et al., 2018; Zoph & Le, 2017; Liu et al., 2018; Pham et al., 2018; Xie et al., 2019; Elsken et al., 2019; Zhou et al., 2019; Baker et al., 2017; Real et al., 2019”