Particle swarm optimization for hyper-parameter selection in deep neural networks

By Pablo Ribalta Lorenzo, Jakub Nalepa, Michal Kawulok, Luciano Sánchez Ramos, José Ranilla. 2017

266 citations recorded by the source corpus.

Deep neural networks (DNNs) have achieved unprecedented success in a wide array of tasks. However, the performance of these systems depends directly on their hyper-parameters which often must be selected by an expert. Optimizing the hyper-parameters remains a substantial obstacle in designing DNNs in practice. In this work, we propose to select them using particle swarm optimization (PSO). Such biologically-inspired approaches have not been extensively exploited for this task. We demonstrate that PSO efficiently explores the solution space, allowing DNNs of a minimal topology to obtain competitive classification performance over the MNIST dataset. We showed that very small DNNs optimized by PSO retrieve promising classification accuracy for CIFAR-10. Also, PSO improves the performance of existing architectures. Extensive experimental study, backed-up with the statistical tests, revealed that PSO is an effective technique for automating hyper-parameter selection and efficiently exploits computational resources.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tong Yu
2. Hong Zhu
3. Wei-Wei Tu
4. Quanming Yao
5. Yang Yu
6. Qiang Yang
7. Yi-Qi Hu
8. Stefan Wermter

Citations in the corpus

• Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “…series of random search methods based on genetic algorithms (Shiffman et al., 2012), such as evolutionary algorithms (Simon, 2013; Orive et al., 2014), particle swarm optimization (Eberhart and Shi, 1998; Lorenzo et al., 2017), and covariance matrix adaption evolutionary strategy (Hansen, 2016).”
• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, methodology): “Among all metaheuristic methods, genetic algorithm (GA) [18] and particle swarm optimization (PSO) [19] are the two most prevalent metaheuristic algorithms used for HPO problems.”
• Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, methodology): “PSO has been used for model selection [53], [91], feature selection for support vector machine (SVM) [91], and hyper-parameter tuning for deep networks [92].”
• Hyperparameter Tuning for Machine Learning Algorithms Used for Arabic Sentiment Analysis (2021, background): “If k parameters with n distinct values are tested, the complexity of Grid Search is expected to grow exponentially at a rate of O(nk) [27].”
• Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks (2018, methodology): “Miikkulainen et al. , 25 Navarro-Guerrero et al. , 26,27 Real et al. , 28 and Xie et al. 29 apply evolutionary algorithms to optimize a subset of hyperparameters for neural networks, while Lorenzo et al. 30 use a particle swarm optimization algorithm.”
• A Survey on Evolutionary Neural Architecture Search (2020, unknown): “SI ACO [105] [135], [137], [158], [159] PSO [73], [74], [96], [101], [102], [109], [130], [160] [37], [38] [24], [46] [55], [84], [161]”