Generalized BackPropagation, Étude De Cas: Orthogonality

By Mehrtash Harandi, Basura Fernando. 2016

64 citations recorded by the source corpus.

This paper introduces an extension of the backpropagation algorithm that enables us to have layers with constrained weights in a deep network. In particular, we make use of the Riemannian geometry and optimization techniques on matrix manifolds to step outside of normal practice in training deep networks, equipping the network with structures such as orthogonality or positive definiteness. Based on our development, we make another contribution by introducing the Stiefel layer, a layer with orthogonal weights. Among various applications, Stiefel layers can be used to design orthogonal filter banks, perform dimensionality reduction and feature extraction. We demonstrate the benefits of having orthogonality in deep networks through a broad set of experiments, ranging from unsupervised feature learning to fine-grained image classification.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Concept whitening for interpretable image recognition (2020, background): “Other work explores ways to learn orthogonal weights or representations for all types of neural networks (not just RNNs) (Harandi and Fernando, 2016; Huang et al., 2018a; Lezcano-Casado and Mart´ınez-Rubio, 2019; Lezama et al., 2018).”
On orthogonality and learning recurrent networks with long term dependencies (2017, background): “Their experimental work indicates that full capacity unitary RNN models can solve the copy memory problem whereas both LSTM networks and restricted capacity unitary RNN models having similar complexity appear unable to solve the task for a longer sequence length (T = 2000).”
Domain Adaptation by Mixture of Alignments of Second-or Higher-Order Scatter Tensors (2016, methodology): “Deep learning [22, 35, 12] has been used in the context of domain adaptation in recent works e.”
Compression-aware Training of Deep Networks (2017, methodology): “This has been achieved by designing new activation functions [5], by explicitly considering the pairwise correlations of the units [54, 37, 40] or of the activations [9, 52], or by constraining the weight matrices of each layer to be orthonormal [21].”
Task-Oriented Feature Distillation (2020, methodology): “propose the generalized backpropagation which can be utilized to train neural layers with orthogonal weights [16].”
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (2019, unknown): “In parallel to the work on unitary RNNs, there has been an increasing interest for optimization over the orthogonal group and the Stiefel manifold in neural networks (Harandi & Fernando, 2016; Ozay & Okatani, 2016; Huang et al., 2017).”