Understanding symmetries in deep networks

By Vijay Badrinarayanan, Bamdev Mishra, Roberto Cipolla. 2015

44 citations recorded by the source corpus.

Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show that a commonly used deep network, which uses convolution, batch normalization, reLU, max-pooling, and sub-sampling pipeline, possess more complex forms of symmetry arising from scaling-based reparameterization of the network weights. We propose to tackle the issue of the weight space symmetry by constraining the filters to lie on the unit-norm manifold. Consequently, training the network boils down to using stochastic gradient descent updates on the unit-norm manifold. Our empirical evidence based on the MNIST dataset shows that the proposed updates improve the test performance beyond what is achieved with batch normalization and without sacrificing the computational efficiency of the weight updates.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Sergey Levine
3. Quoc V. Le
4. Kyunghyun Cho
5. Razvan Pascanu
6. Stefano Ermon
7. Mohit Bansal
8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yoshua Bengio
3. Dacheng Tao
4. Trevor Darrell
5. Ming-Hsuan Yang
6. Xiaogang Wang
7. Jitendra Malik
8. Andrew Zisserman

In Model Architectures

1. Yoshua Bengio
2. Samy Bengio
3. Razvan Pascanu
4. Laurent Dinh
5. Jascha Narain Sohl-Dickstein
6. Kyunghyun Cho
7. Quoc V. Le
8. J. Urgen Schmidhuber

In Scaling & Learning Theory

1. Yoshua Bengio
2. Razvan Pascanu
3. Jascha Narain Sohl-Dickstein
4. Samy Bengio
5. F. Sultana
6. Paramartha Dutta
7. Laurent Dinh
8. Abu Sufian

Citations in the corpus

• Density estimation using Real NVP (2016, methodology): “To further improve the propagation of training signal, we use deep residual networks [24, 25] with batch normalization [31] and weight normalization [2, 54] in s and t.”
• Sharp Minima Can Generalize For Deep Nets (2017, methodology): “Instances of commonly used reparametrization are batch normalization (Ioffe & Szegedy, 2015), or the virtual batch normalization variant (Salimans et al., 2016), and weight normalization (Badrinarayanan et al., 2015; Salimans & Kingma, 2016; Arpit et al., 2016).”
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “h layer in the encoder network, a convolutional operation is performed using lter bank to produce feature maps. Then, to reduce internal covariate shift the authors have used batch normalization [98] [99] followed by ReLU [100] nonlinearity operation. Resulting output feature maps are max-pooled usi”