Backprop Evolution

By Maximilian Alber, Irwan Bello, Barret Zoph, Pieter-Jan Kindermans, Prajit Ramachandran, Quoc V. Le. 2018

9 citations recorded by the source corpus.

The back-propagation algorithm is the cornerstone of deep learning. Despite its importance, few variations of the algorithm have been attempted. This work presents an approach to discover new variations of the back-propagation equation. We use a domain specific lan- guage to describe update equations as a list of primitive functions. An evolution-based method is used to discover new propagation rules that maximize the generalization per- formance after a few epochs of training. We find several update equations that can train faster with short training times than standard back-propagation, and perform similar as standard back-propagation at convergence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

Citations in the corpus

• Attention Augmented Convolutional Networks (2019, neutral): “utomated architecture search procedures proves useful to find even better models than those previously found in image classification [52], object detection [12], image segmentation [6] and other domains[5,1,33,8]. Finally, one can ask to which degree fully attentional models can replace convolutional ”