Neural networks designing neural networks: Multi-objective hyper-parameter optimization

By Sean C. Smithson, Guang Yang, Warren J. Gross, B. Meyer. 2016

104 citations recorded by the source corpus.

Artificial neural networks have gone through a recent rise in popularity, achieving state-of-the-art results in various fields, including image classification, speech recognition, and automated control. Both the performance and computational complexity of such models are heavily dependant on the design of characteristic hyper-parameters (e.g., number of hidden layers, nodes per layer, or choice of activation functions), which have traditionally been optimized manually. With machine learning penetrating low-power mobile and embedded areas, the need to optimize not only for performance (accuracy), but also for implementation complexity, becomes paramount. In this work, we present a multi-objective design space exploration method that reduces the number of solution networks trained and evaluated through response surface modelling. Given spaces which can easily exceed 1020 solutions, manually designing a near-optimal architecture is unlikely as opportunities to reduce network complexity, while maintaining performance, may be overlooked. This problem is exacerbated by the fact that hyper-parameters which perform well on specific datasets may yield sub-par results on others, and must therefore be designed on a per-application basis. In our work, machine learning is leveraged by training an artificial neural network to predict the performance of future candidate networks. The method is evaluated on the MNIST and CIFAR-10 image datasets, optimizing for both recognition accuracy and computational complexity. Experimental results demonstrate that the proposed method can closely approximate the Pareto-optimal front, while only exploring a small fraction of the design space.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Neural Architecture Search (2019, background): “the cheap-to-evaluate objectives in order to select candidates from low-density regions. Finally, the expensive-toevaluate objectives for the candidates are evaluated and the Pareto front is updated. Smithson et al. (2016) Smithson et al. (2016) propose a surrogate model-based approach to search for”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, methodology): “, kernel sizes,etc.). 2 It is worth noting that existing methodologies rely on simplistic proxies of the hardware performance (e.g., counts of the NN’s weights [8] or energy per operation assumptions [22, 25]), or simulation results [21]. Since our target application is the mobile and embedded desig”
DLOPT: Deep Learning Optimization Library (2018, methodology): “Therefore, it seems mandatory to use an automatic intelligent tool to ﬁnd an efﬁcient hyper-parameter conﬁguration (e.g., grid, evolutionary or random search) 4–6 .”
Random error sampling-based recurrent neural network architecture optimization (2019, background): “They require to ﬁt a model and to evaluate its performance on validation data (i.e., they are data-driven), which can be a demanding process (time and computational resources) [6, 18, 19].”
A Specialized Evolutionary Strategy Using Mean Absolute Error Random Sampling to Design Recurrent Neural Networks (2019, background): “, they are data-driven), which can be an expensive process [18], [6], [19].”
Low-Cost Recurrent Neural Network Expected Performance Evaluation (2018, background): “Mainly because (in general) these methods are data driven, and therefore, they require fitting a model and evaluating its performance on validation data, which can be an expensive process [2, 6, 24].”
NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks (2017, methodology): “Smithsons et al. (2016) have used a pair of trainer-trainee CNNs to incorporate accuracy and cost into the design of neural networks.”
HyperPower: Power- and memory-constrained hyper-parameter optimization for neural networks (2017, methodology): “Random (model-free) methods : Random [5] and random-walk [8] hyper-parameter selection has been shown to perform well in problems that have low effective dimensionality [1].”