Training Deep Networks with Structured Layers by Matrix Backpropagation

By Catalin Ionescu, Orestis Vantzos, Cristian Sminchisescu. 2015

92 citations recorded by the source corpus.

Deep neural network architectures have recently produced excellent results in a variety of areas in artificial intelligence and visual recognition, well surpassing traditional shallow architectures trained using hand-designed features. The power of deep networks stems both from their ability to perform local computations followed by pointwise non-linearities over increasingly larger receptive fields, and from the simplicity and scalability of the gradient-descent training procedure based on backpropagation. An open problem is the inclusion of layers that perform global, structured matrix computations like segmentation (e.g. normalized cuts) or higher-order pooling (e.g. log-tangent space metrics defined over the manifold of symmetric positive definite matrices) while preserving the validity and efficiency of an end-to-end deep training framework. In this paper we propose a sound mathematical apparatus to formally integrate global structured computation into deep computation architectures. At the heart of our methodology is the development of the theory and practice of backpropagation that generalizes to the calculus of adjoint matrix variations. The proposed matrix backpropagation methodology applies broadly to a variety of problems in machine learning or computational perception. Here we illustrate it by performing visual segmentation experiments using the BSDS and MSCOCO benchmarks, where we show that deep networks relying on second-order pooling and normalized cuts layers, trained end-to-end using matrix backpropagation, outperform counterparts that do not take advantage of such global layers.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

COCO-Stuff: Thing and Stuff Classes in Context (2016, background): “One reason for the recent success of deep learning methods is the advent of large-scale datasets [43, 29, 63].”
Learning Monocular 3D Human Pose Estimation from Multi-view Images (2018, background): “quire a singular value decomposition and is intricate to differentiate [9].”
Building Deep Networks on Grassmann Manifolds (2016, methodology): “Following [34] to deal with the case of outputting a tuple, we apply the new chain rule Eqn.9 with the equations a : F ( b ) = F ∗ ( a ) : b and d X k = F ( d X k − 1 ) to achieve the update rule for the data: where the two variations d Q and d R are derived by the variation of the QR operation d X…”
Decorrelated Batch Normalization (2018, methodology): “We achieve this by using the fact that eigen-decomposition is differentiable, and its derivatives can be obtained using matrix differential calculus, as shown by prior work [15, 26].”
A Riemannian Network for SPD Matrix Learning (2016, methodology): “While (Ionescu, Vantzos, and Sminchisescu 2015) introduced a covariance pooling layer into ConvNets starting from images, our SPDNet works on SPD matrices directly and exploits multiple layers tailored for SPD matrix deep learning.”
Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization (2017, methodology): “The gradients associated with the structured layers are derived using matrix backpropagation methodology [13], which establishes the chain rule of a general matrix function by first-order Taylor approximation.”
Multi-view Harmonized Bilinear Network for 3D Object Recognition (2018, background): “The readers can refer to [13] for detailed derivation of Eq.”
Cascaded Projection: End-To-End Network Compression and Acceleration (2019, methodology): “[23] impose orthogonality constraints on the weights of a neural network during training using a method for backpropagation of gradients through structured linear algebra layers developed in [26, 27].”
Matrix Backpropagation for Deep Networks with Structured Layers (2015, methodology): “We provide detailed derivations (see also our extended version[22]), operating conditions for spectral and non-linear layers, and illustrate the methodology for normalized cuts and second-order pooling layers.”
Deep Metric Learning via Facility Location (2016, background): “To overcome this, matrix backpropagation [12] first projects the true and predicted affinity matrix to a metric space where Euclidean distance is appropriate.”
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (2017, methodology): “To achieve this, we use the result from matrix differential calculus [14], which combines the derivatives of eigenvalues and eigenvectors based on chain rule: given ∂ L ∂ D ∈ R n × n and ∂ L ∂ Λ ∈ R n × n , where L is the loss function, the back-propagate derivatives are and ( ∂ L ∂ Λ ) diag sets…”
DeepVCP: An End-to-End Deep Neural Network for Point Cloud Registration (2019, unknown): “Thanks to [18], the latest Tensorflow has supported the SVD operator and its backpropagation.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “This can be achieved by using matrix differential calculus [78], as proposed in [34].”
Is Second-Order Information Helpful for Large-Scale Visual Recognition? (2017, methodology): “As MPN-COV involves nonlinear matrix functions whose backpropagation is not straightforward, we develop the gradients associated with MPN-COV based on the matrix propagation methodology [12] for end-to-end learning.”