Investigating recurrent neural network memory structures using neuro-evolution

By Alexander Ororbia, A. Elsaid, Travis Desell. 2019

62 citations recorded by the source corpus.

This paper presents a new algorithm, Evolutionary eXploration of Augmenting Memory Models (EXAMM), which is capable of evolving recurrent neural networks (RNNs) using a wide variety of memory structures, such as Δ-RNN, GRU, LSTM, MGU and UGRNN cells. EXAMM evolved RNNs to perform prediction of large-scale, real world time series data from the aviation and power industries. These data sets consist of very long time series (thousands of readings), each with a large number of potentially correlated and dependent parameters. Four different parameters were selected for prediction and EXAMM runs were performed using each memory cell type alone, each cell type and simple neurons, and with all possible memory cell types and simple neurons. Evolved RNN performance was measured using repeated k-fold cross validation, resulting in 2420 EXAMM runs which evolved 4, 840, 000 RNNs in ~24,200 CPU hours on a high performance computing cluster. Generalization of the evolved RNNs was examined statistically, providing findings that can help refine the design of RNN memory cells as well as inform future neuro-evolution algorithms.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, methodology): “Besides NEAT, there are several evolutionary-based approaches to evolve an RNN, such as EXALT [14], EXAMM [32], or a method using ant colony optimization (ACO) to improve LSTM RNNs by refining their cellular structures [15].”
Random error sampling-based recurrent neural network architecture optimization (2019, methodology): “To test our proposal and answer the research questions we selected three problems: the sine wave, the waste generation prediction problem [41], and the coal-ﬁred power plant ﬂame intensity prediction problem [38].”