Evolutionary neural AutoML for deep learning

By Jason Zhi Liang, Elliot Meyerson, Babak Hodjat, Daniel Fink, Karl Mutch, Risto Miikkulainen. 2019

120 citations recorded by the source corpus.

Deep neural networks (DNNs) have produced state-of-the-art results in many benchmarks and problem domains. However, the success of DNNs depends on the proper configuration of its architecture and hyperparameters. Such a configuration is difficult and as a result, DNNs are often not used to their full potential. In addition, DNNs in commercial applications often need to satisfy real-world design constraints such as size or number of parameters. To make configuration easier, automatic machine learning (AutoML) systems for deep learning have been developed, focusing mostly on optimization of hyperparameters. This paper takes AutoML a step further. It introduces an evolutionary AutoML framework called LEAF that not only optimizes hyperparameters but also network architectures and the size of the network. LEAF makes use of both state-of-the-art evolutionary algorithms (EAs) and distributed computing frameworks. Experimental results on medical image classification and natural language analysis show that the framework can be used to achieve state-of-the-art performance. In particular, LEAF demonstrates that architecture optimization provides a significant boost over hyperparameter optimization, and that networks can be minimized at the same time with little drop in performance. LEAF therefore forms a foundation for democratizing and improving AI, as well as making AI practical in future applications.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yang Liu
2. Qiang Yang
3. Tianjian Chen
4. Radwa El Shawi
5. Sherif Sakr
6. Mohamed Maher
7. Xinle Liang
8. Fábio Pinto

Citations in the corpus

• Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “LEAF consists of three main layers which are algorithm layers, system layer and problem-domain layer.”
• Automatic Model Monitoring for Data Streams (2019, methodology): “Recently, there has been a growing interest in researching autoML methods [4, 13, 19] – the sub-field of ML research that focuses on developing methods with the aim of automating different stages”
• A Survey on Evolutionary Neural Architecture Search (2020, unknown): “Other Memetic [134], [162] DE [48], [74] [89] [51] [163] HCA [31], [58], [79] CVOA [87] Hyper-heuristic [77] FA [76] AIS [75] Multiobjective EA [64], [72], [80], [100], [104], [106], [114], [114], [120], [128], [133], [136], [164]–[167] [54], [168] [169] [170], [171]”
• Self-supervised Cross-silo Federated Neural Architecture Search (2021, background): “RL-based or evolutionary algorithm based NAS techniques may consume more than 1000 GPU days to achieve the state-of-the-art results [38], [42].”