MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes
By Henry Gouk, Bernhard Pfahringer, Eibe Frank, Michael J. Cree. 2018
7 citations recorded by the source corpus.
Effective regularisation of neural networks is essential to combat overfitting due to the large number of parameters involved. We present an empirical analogue to the Lipschitz constant of a feed-forward neural network, which we refer to as the maximum gain. We hypothesise that constraining the gain of a network will have a regularising effect, similar to how constraining the Lipschitz constant of a network has been shown to improve generalisation. A simple algorithm is provided that involves rescaling the weight matrix of each layer after each parameter update. We conduct a series of studies on common benchmark datasets, and also a novel dataset that we introduce to enable easier significance testing for experiments using convolutional networks. Performance on these datasets compares favourably with other common regularisation techniques.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hanie Sedghi
- Philip M. Long
- Vineet Gupta
- David P. Helmbold
- Behnam Neyshabur
- Qiang Yang
- Yu Zhang
- Koby Crammer
In Computer Vision and Pattern Recognition
In Statistical and Nonlinear Physics
Citations in the corpus
- The Singular Values of Convolutional Layers (2018, background): “Up until now, authors seeking to control the operator norm of convolutional layers have resorted to approximations (Yoshida and Miyato, 2017; Miyato et al., 2018; Gouk et al., 2018a).”