Evolving Unsupervised Deep Neural Networks for Learning Meaningful Representations

By Yanan Sun, Gary G. Yen, Zhang Yi. 2017

139 citations recorded by the source corpus.

Deep learning (DL) aims at learning the meaningful representations. A meaningful representation gives rise to significant performance improvement of associated machine learning (ML) tasks by replacing the raw data as the input. However, optimal architecture design and model parameter estimation in DL algorithms are widely considered to be intractable. Evolutionary algorithms are much preferable for complex and nonconvex problems due to its inherent characteristics of gradient-free and insensitivity to the local optimal. In this paper, we propose a computationally economical algorithm for evolving unsupervised deep neural networks to efficiently learn meaningful representations, which is very suitable in the current big data era where sufficient labeled data for training is often expensive to acquire. In the proposed algorithm, finding an appropriate architecture and the initialized parameter values for an ML task at hand is modeled by one computational efficient gene encoding approach, which is employed to effectively model the task with a large number of parameters. In addition, a local search strategy is incorporated to facilitate the exploitation search for further improving the performance. Furthermore, a small proportion labeled data is utilized during evolution search to guarantee the learned representations to be meaningful. The performance of the proposed algorithm has been thoroughly investigated over classification tasks. Specifically, error classification rate on MNIST with 1.15% is reached by the proposed algorithm consistently, which is considered a very promising result against state-of-the-art unsupervised DL algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yaochu Jin
2. Yanan Sun
3. Mengjie Zhang
4. Bing Xue
5. Gary G. Yen
6. Xing-yi Zhang
7. Ye Tian
8. Handing Wang

In Computer Vision and Pattern Recognition

1. Bing Xue
2. Mengjie Zhang
3. Yanan Sun
4. Gary G. Yen
5. Yaochu Jin
6. Handing Wang
7. Yuqiao Liu
8. Kate Saenko

Citations in the corpus

• Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, background): “Unfortunately, designing architecture with the best performance for the investigated data requires extensive expertise in both the CNNs and the data domain [7], which is not necessarily held by the interested users.”
• A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification (2018, background): “Sincethe difficulties of manually designing the architectures of CNNs have been raised more frequently in recent years, exploiting evolutionary computation (EC) algorithms to generate deep neural networks [18] has come into the spot-light to resolve the issues.”
• Completely Automated CNN Architecture Design Based on Blocks (2020, methodology): “…the heuristic computational paradigms [9], such as random search [10], Bayesian-based Gaussian process [11], [12], tree-structured Parzen estimators [13], sequential model-based global optimization [14], neuroevolution of augmenting topologies [15], and evolutionary unsupervised deep learning [8].”
• A Survey on Evolutionary Neural Architecture Search (2020, unknown): “objective EA GAs [9], [19]–[23], [33]–[35], [47], [49], [52], [56], [59]–[61], [63], [65]–[67], [69]–[71], [74], [78], [81], [82], [85], [86], [88], [90], [92], [93], [95], [97]– [99], [103], [107], [110], [113], [117]– [119], [121], [122], [125]–[127], [131], [138], [143], [144] [39], [145] [44], [”
• An Evolutionary Algorithm for Large-Scale Sparse Multiobjective Optimization Problems (2020, background): “…et al. [28] made use of appropriate greedy rules in genetic operators to improve the quality of offsprings for critical node detection, and Sun et al. [48] assumed a set of orthogonal vectors to highly reduce the number of decision variables in training an unsupervised deep neural network.”
• Evolving Deep Convolutional Neural Networks by Variable-Length Particle Swarm Optimization for Image Classification (2018, unknown): “Interested researchers have done excellent work on the automatic design of the CNN architectures by using Genetic Programming (GP) [8] and Genetic Algorithms (GAs) [9][10], such as Large-scale evolution of image classifiers (LEIC) method [11] recently proposed by Google, which have shown that EC can”