Reducing Overfitting in Deep Networks by Decorrelating Representations

By Michael Cogswell, Faruk Ahmed, Ross B. Girshick, C. Lawrence Zitnick, Dhruv Batra. 2015

461 citations recorded by the source corpus.

One major challenge in training Deep Neural Networks is preventing overfitting. Many techniques such as data augmentation and novel regularizers such as Dropout have been proposed to prevent overfitting without requiring a massive amount of training data. In this work, we propose a new regularizer called DeCov which leads to significantly reduced overfitting (as indicated by the difference between train and val performance), and better generalization. Our regularizer encourages diverse or non-redundant representations in Deep Neural Networks by minimizing the cross-covariance of hidden activations. This simple intuition has been explored in a number of past works but surprisingly has never been applied as a regularizer in supervised learning. Experiments across a range of datasets and network architectures show that this loss always reduces overfitting while almost always maintaining or increasing generalization performance and often improving performance over Dropout.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving Neural Language Modeling via Adversarial Training (2019, methodology): “…method is that it provides an effective mechanism to encourage diversity of word embedding vectors, which is widely observed to yield better generalization performance in neural language models (e.g., Mu et al., 2018; Gao et al., 2019; Liu et al., 2018b; Cogswell et al., 2016; Khodak et al., 2018).”
Concept whitening for interpretable image recognition (2020, methodology): “Other methods perform whitening by adding a decorrelation loss (Cogswell et al., 2016).”
Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning (2021, methodology): “For example, [3, 14, 35] rely on feature decorrelation to solve the mode collapse problem in self-supervised learning when negative pairs are missing, and [5,15,34] use decorrelation to improve neural networks’ features and hence boost generalization.”
Selfless Sequential Learning (2018, methodology): “- Decov (Cogswell et al., 2015) aims at reducing overfitting by decorrelating neuron activations.”
MultiGrain: a unified image embedding for classes and instances (2019, positive): “We observed that inducing decorrelation via a loss [5] is insufficient to ensure that features generalize well, which concurs with prior works [11, 34].”
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups (2016, background): “[21] observe a correlation between the covariance of hidden unit activations and overfitting.”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, background): “DeCov proposed by Cogswell et al. (2015) tries to minimize the off-diagonal terms of the sample covariance matrix of activations, thus encouraging the activations to be as decorrelated as possible.”
Learning towards Minimum Hyperspherical Energy (2018, background): “Another stream of works seeks to further release the network generalization power by alleviating redundancy through diversification [57, 56, 5, 36] as rigorously analyzed by [59].”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, background): “Here, the convolution kernels are set to be independent to each other by initializing the weight tensors as unitary matrices for better generalization and fast learning (Cogswell et al., 2015).”
NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES (2018, background): “Another known neural network property is that the responses of a layer exhibit considerable correlation (Denil et al., 2013), inspiring the idea of learning decorrelated ﬁlters (Cogswell et al., 2016; Rodríguez et al., 2017).”
Regularizing Neural Networks via Minimizing Hyperspherical Energy (2019, background): “[28, 29, 30, 9, 8, 30] characterize diversity among neurons with orthogonality, and regularize the neural network by promoting the orthogonality.”
Low-Shot Visual Recognition by Shrinking and Hallucinating Features (2016, background): “Another form of feature regularization involves minimizing the correlation between the features [7, 8].”
Skip Connections Eliminate Singularities (2017, background): “More complicated joint priors over hidden unit responses that favor decorrelated (Cogswell et al., 2015) or clustered (Liao et al., 2016) responses have been proposed before.”
NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks (2020, background): “Intuitively speaking, the design of traditional CNNs is ad-hoc and introduces redundancy [29, 4].”
Dual Attention Matching Network for Context-Aware Feature Sequence Based Person Re-identification (2018, background): “In [9], de-correlating representations has been studied as a regularizer to reduce over-ﬁtting in deep networks.”
On Feature Decorrelation in Self-Supervised Learning (2021, background): “Meanwhile, feature decorrelation appears to be a valuable idea in the field of machine learning: In discriminative tasks, [12, 48] introduce in objective functions additional terms regularizing correlation matrices and [25, 26] develop normalization layers standardizing covariance matrices to obtain”
Learning Neural Networks with Adaptive Regularization (2019, methodology): “, weight decay [26], early stopping [5], Dropout [39] and the more recent DeCov [6] method.”
Deep Complex Networks (2017, background): “This has the advantage of avoiding co-adaptation between the two components which reduces the risk of overfitting (Cogswell et al., 2015; Srivastava et al., 2014).”