Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint

By Wei Xiong, Bo Du, Lefei Zhang, RuiMin Hu, Dacheng Tao. 2016

40 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “For example, [3, 14, 35] rely on feature decorrelation to solve the mode collapse problem in self-supervised learning when negative pairs are missing, and [5,15,34] use decorrelation to improve neural networks’ features and hence boost generalization.”
Selfless Sequential Learning (2018, methodology): “Note that this is somewhat more ﬂexible than decorrelating neurons in ﬁxed groups as used in Xiong et al. (2016).”
On Feature Decorrelation in Self-Supervised Learning (2021, background): “Meanwhile, feature decorrelation appears to be a valuable idea in the field of machine learning: In discriminative tasks, [12, 48] introduce in objective functions additional terms regularizing correlation matrices and [25, 26] develop normalization layers standardizing covariance matrices to obtain”
Compression-aware Training of Deep Networks (2017, methodology): “This has been achieved by designing new activation functions [5], by explicitly considering the pairwise correlations of the units [54, 37, 40] or of the activations [9, 52], or by constraining the weight matrices of each layer to be orthonormal [21].”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “other hand, previous works suggest that further decorrelating the activations is beneficial to both the optimization [9, 29] and generalization [6, 49].”
Continual Learning in Neural Networks (2019, methodology): “Note that this is somewhat more flexible than decorrelating neurons in fixed groups as used in [158].”
Understanding of a convolutional neural network (2017, background): “Thus, if we want to have 32×32 neurons in the next layer, then we will have 5×5×3 by 32x32 connections which is 76,800 connections (compared to 3,145,728 for full connectivity), as in [1,9,13,14,17].”
Decorrelated Batch Normalization (2018, methodology): “[53] extends [9] to learn group-wise decorrelated representations.”