Evolving Memory Cell Structures for Sequence Learning

By Justin Bayer, Daan Wierstra, Julian Togelius, J. Urgen Schmidhuber. 2009

106 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “It is possible to evolve good problem-specific topologies (Bayer et al., 2009).”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Other AutoML studies similarly have found ways to constrain their search spaces to isolated algorithmic aspects, such as the learning rule used during backpropagation [2, 67], the gating structure of an LSTM [6, 34], or the data augmentation [17, 62]; in these works, all other algorithmic aspects re”
Evolving Deep Neural Networks (2017, background): “variations of individual LSTM memory unit architecture [4, 7, 25, 28], and discovering new ways of stitching LSTM layers into a network [8, 26, 53].”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “Early attempts used evolutionary algorithms to learn the topology of LSTM cells [128], while later approaches leveraged RL to generate descriptions for good CNN architec-”
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder (2016, positive): “Because many amazing results were achieved by LSTM, RNN has widely been equipped with LSTM units [25, 26, 27, 28, 22, 29, 30].”
An Empirical Exploration of Recurrent Network Architectures (2015, methodology): “Architecture search for RNNs has been previously conducted by Bayer et al. (2009). They attempted to address the same problem, but they have done many fewer experiments, with small models (5 units). They only considered synthetic problems with long term dependencies, and they were able to find archi”
Understanding and Simplifying One-Shot Architecture Search (2018, methodology): “Genetic and neuro-evolution algorithms have also been used for designing good neural network architectures, e.g., Stanley & Miikkulainen (2002); Bayer et al. (2009); Jozefowicz et al. (2015); Miikkulainen et al. (2017); Xie & Yuille (2017).”
Evolving Deep LSTM-based Memory Networks using an Information Maximization Objective (2016, methodology): “Bayer et al.[5] evolved custom LSTM memory cells using mutation operators.”
Large-Scale Evolution of Image Classifiers (2017, background): “Tangentially, there has also been neuro-evolution work on LSTM structure (Bayer et al., 2009; Zaremba, 2015), but this is beyond the scope of this paper.”
A Critical Review of Recurrent Neural Networks for Sequence Learning (2015, background): “Several recent papers explore genetic algorithms for neural networks, especially as means of learning the architecture of neural networks, a problem not addressed by backpropagation [Bayer et al., 2009, Harp and Samad, 2013].”
Discovering Gated Recurrent Neural Network Architectures (2018, methodology): “Recent research in LSTMs has been broadly focused in two directions - first, in finding variations of individual LSTM memory unit architecture (Bayer et al. (2009b), Jozefowicz et al.”
A review on the long short-term memory model (2020, background): “In the work of Bayer et al. (2009), the search for improvements of the model was already ongoing.”
Squeeze-and-Excitation Networks (2017, background): “ing good memory cells for sequence models [27], [28] and learning sophisticated architectures for large-scale image classification [29], [30], [31].”
Supervised Sequence Labelling with Recurrent Neural Networks (2012, background): “Indeed it has been shown that alternative structures with equally good performance on toy problems such as learning contextfree and context-sensitive languages can be evolved automatically (Bayer et al., 2009).”
Understanding LSTM - a tutorial into Long Short-Term Memory Recurrent Neural Networks (2019, unknown): “These partially problem speciﬁc and can be derived [3] from the basic method [41], [21] covered in Section 8 and 9.”
Increasing generality in machine learning through procedural content generation (2019, background): “We identify ﬁve main open challenges that we believe are essential in pushing the ﬁeld of PCG forward and to realizing its promise to create more adaptive and lifelong learning ML agents.”