OLE: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning
By José Lezama, Qiang Qiu, Pablo Musé, Guillermo Sapiro. 2017
88 citations recorded by the source corpus.
Deep neural networks trained using a softmax layer at the top and the cross-entropy loss are ubiquitous tools for image classification. Yet, this does not naturally enforce intra-class similarity nor inter-class margin of the learned deep representations. To simultaneously achieve these two goals, different solutions have been proposed in the literature, such as the pairwise or triplet losses. However, these carry the extra task of selecting pairs or triplets, and the extra computational burden of computing and learning for many combinations of them. In this paper, we propose a plug-and-play loss term for deep networks that explicitly reduces intra-class variance and enforces inter-class margin simultaneously, in a simple and elegant geometric manner. For each class, the deep features are collapsed into a learned linear subspace, or union of them, and inter-class subspaces are pushed to be as orthogonal as possible. Our proposed Orthogonal Low-rank Embedding (OLÉ) does not require carefully crafting pairs or triplets of samples for training, and works standalone as a classification loss, being the first reported deep metric learning framework of its kind. Because of the improved margin between features of different classes, the resulting deep networks generalize better, are more discriminative, and more robust. We demonstrate improved classification performance in general object recognition, plugging the proposed loss term into existing off-the-shelf architectures. In particular, we show the advantage of the proposed loss in the small data/model scenario, and we significantly advance the state-of-the-art on the Stanford STL-10 benchmark.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- A Real-Time and Hardware-Efficient Processor for Skeleton-Based Action Recognition With Lightweight Convolutional Neural Network (2019, methodology): “(10) Using the training strategy of the orthogonal low-rank embedding (OLE) [20], the singular values in the penultimate layer are centralized with near 15% largest singular values having near 70% retained information that is defined as Tr( k×k)Tr( n×n) .”
- FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, background): “, 2015), have almost orthogonal rows because the orthogonality of two random vectors is observed in high dimensions with high probability (Lezama et al., 2018).”
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Orthogonal regularization methods have demonstrated improved performance in image classification [159], [161], [162], resisting attacks from adversarial examples [163], neural photo editing [164] and training GANs [25], [32].”
- Orthogonal Projection Loss (2021, unknown): “• Our orthogonality constraints are efficiently formulated compared to existing methods [27, 48], allowing mini-batch processing without the need to explicitly obtain singular values.”