Use of genetic programming for the search of a new learning rule for neural networks

By Samy Bengio, Yoshua Bengio, Jocelyn Cloutier. 1994

In previous work we explained how to use standard optimization methods such as simulated annealing, gradient descent and genetic algorithms to optimize a parametric function which could be used as a learning rule for neural networks. To use these methods, we had to choose a fixed number of parameters and a rigid form for the learning rule. In this article, we propose to use genetic programming to find not only the values of rule parameters but also the optimal number of parameters and the form of the rule. Experiments on classification tasks suggest genetic programming finds better learning rules than other optimization methods. Furthermore, the best rule found with genetic programming outperformed the well-known backpropagation algorithm for a given set of tasks.<<ETX>>

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Neural Optimizer Search with Reinforcement Learning (2017, methodology): “, Bengio et al. (1994); Runarsson & Jonsson (2000); Orchard & Wang (2016)).”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “[8] is that they also use simple mathematical operations as building blocks.”
Evolving Reinforcement Learning Algorithms (2021, methodology): “Genetic programming has been used to find new loss functions (Bengio et al., 1994; Trujillo & Olague, 2006).”
Backprop Evolution (2018, background): “A few attempts have been made to change the back-propagation equation with some degrees of success (e.g., Bengio et al. (1994); Lillicrap et al. (2014); Lee et al. (2015); Nøkland (2016); Liao et al. (2016)).”