HyperPower: Power- and memory-constrained hyper-parameter optimization for neural networks

By Dimitrios Stamoulis, Ermao Cai, Da-Cheng Juan, Diana Marculescu. 2017

77 citations recorded by the source corpus.

While selecting the hyper-parameters of Neural Networks (NNs) has been so far treated as an art, the emergence of more complex, deeper architectures poses increasingly more challenges to designers and Machine Learning (ML) practitioners, especially when power and memory constraints need to be considered. In this work, we propose HyperPower, a framework that enables efficient Bayesian optimization and random search in the context of power- and memory-constrained hyperparameter optimization for NNs running on a given hardware platform. HyperPower is the first work (i) to show that power consumption can be used as a low-cost, a priori known constraint, and (ii) to propose predictive models for the power and memory of NNs executing on GPUs. Thanks to HyperPower, the number of function evaluations and the best test error achieved by a constraint-unaware method are reached up to 112.99× and 30.12× faster, respectively, while never considering invalid configurations. HyperPower significantly speeds up the hyper-parameter optimization, achieving up to 57.20× more function evaluations compared to constraint-unaware methods for a given time interval, effectively yielding significant accuracy improvements by up to 67.6%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours (2019, methodology): “Hardware-aware NAS: Earlier NAS methods focused on maximizing accuracy under FLOPs constraints [22,25], but low FLOP count does not necessarily translate to hardware efficiency [10,18].”
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, background): “This helps increase the prediction conﬁdence level of the GP regressor over the entire adaptation search space [27].”
Designing Adaptive Neural Networks for Energy-Constrained Image Classification (2018, methodology): “Net hyper-parameters [14]. Each network is trained using its original learning hyper-parameters (e.g., learning rate), but these can be also treated as variables to solve for using Algorithm 1, as in [27]. 2This visualization is insightful since the constrained optimization problem under considerati”
MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, methodology): “tematic accelerator performance analysis and optimization methods [4, 23, 46, 58, 59, 63, 67, 68].”
A systematic review of Green AI (2023, background): “Hence, most publications within this topic focus on identifying alternative strategies that reduce the number of iterations required to tune hyper-parameters (Stamoulis et al., 2018a).”
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization (2019, methodology): “Hardware-aware NAS : Earlier NAS methods focused on maximizing accuracy under FLOPs constraints [19], [28], but low FLOP count does not necessarily translate to hardware efﬁciency [29], [30].”
FLightNNs: Lightweight Quantized Deep Neural Networks for Fast and Accurate Inference (2019, methodology): “To reduce the model size, previous work has also conducted neural architecture search with energy constraint [21, 23, 24, 28].”
Estimation of energy consumption in machine learning (2019, methodology): “Approaches to automate the process of designing energy-aware neural network models use techniques such as reinforcement learning [74], Bayesian optimization [71] and genetic algorithms [15].”
Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models (2020, methodology): “Energy efﬁciency of inference in deep neural networks (DNNs) is also an active area of research with methods such as quantization aware training, energy-aware pruning (Yang et al., 2017), and power-and memory-constrained hyperparameter optimization like HyperPower (Stamoulis et al., 2018).”