Unsupervised Embedding Learning via Invariant and Spreading Instance Feature

By Mang Ye, Xu Zhang, Pong Chi Yuen, Shih-Fu Chang. 2019

642 citations recorded by the source corpus.

This paper studies the unsupervised embedding learning problem, which requires an effective similarity measurement between samples in low-dimensional embedding space. Motivated by the positive concentrated and negative separated properties observed from category-wise supervised learning, we propose to utilize the instance-wise supervision to approximate these properties, which aims at learning data augmentation invariant and instance spread-out features. To achieve this goal, we propose a novel instance based softmax embedding method, which directly optimizes the `real' instance features on top of the softmax function. It achieves significantly faster learning speed and higher accuracy than all existing methods. The proposed method performs well for both seen and unseen testing categories with cosine similarity. It also achieves competitive performance even without pre-trained network over samples from fine-grained categories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Simple Framework for Contrastive Learning of Visual Representations (2020, neutral): “Other work explores the use of in-batch samples for negative sampling instead of a memory bank (Ye et al., 2019; Ji et al., 2019).”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “Following [63, 2], we take two random “views” of the same image under random data augmentation to form a positive pair.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “More recent approaches employ the Noise Contrastive Estimation [49] learning paradigm, and use contrastive losses [50] which encourage similarity between image features under different random transformations on single input image [51].”
Improved Baselines with Momentum Contrastive Learning (2020, background): “1a) [13, 8, 17, 1, 9, 2], the negative keys are from the same batch and updated end-to-end by back-propagation.”
Contrastive Multiview Coding (2019, background): “[26, 81, 72, 25, 31, 57, 3, 29, 36] that take advantage of the co-occurence of multiple views.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “Following [45, 21, 2, 9], in MoCo v3 we use the keys that naturally co-exist in the same batch.”
Exploring Simple Siamese Representation Learning (2020, background): “contrastive learning [16] is to attract the positive sample pairs and repulse the negative sample pairs. This methodology has been recently popularized for un-/self-supervised representation learning [36,30,20,37,21,2,35,17,29,8,9]. Simple and effective instantiations of contrastive learning have be”
Self-Supervised Learning of Pretext-Invariant Representations (2019, neutral): “Similar to our work, recent methods also focus on invariance [47] or decoupling the pretext task [17] to learn representations.”
What makes for good views for contrastive learning (2020, background): “For example: luminance and chrominance decomposition [53], randomly augmenting an image twice [58, 7, 4, 23, 60, 50, 65, 67], using different time-steps of videos [42, 66, 46, 22, 21], patches of the same image [27, 42, 26], multiple sensory data [39, 9, 44], or text and its context [37, 59, 35, 32]”
Prototypical Contrastive Learning of Unsupervised Representations (2020, methodology): “The end-to-end approach [2, 7, 9] generates instance features using all samples within the current mini-batch, and apply the same encoder to both the original samples and their augmented version.”
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations (2021, background): “Many works use a memory bank to store the instance class representation vector [67, 77, 59, 23, 44, 9], while some others explore the usage of in-batch negative samples instead [14, 71, 28, 31].”
Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams (2020, methodology): “losses, whereas other works use batch information in lifted structure embeddings [35] or instance-wise softmax embeddings [51].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “The recent trend of contrastive learning [44, 19, 106, 41, 45, 90] integrates data augmentations and instance discrimination by maintaining relative consistency between representations of an image and its augmented view.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “Discriminative self-supervised methods [14, 8] are based on direct comparison between training samples, and include contrastive methods where each sample in the dataset is considered as its own class [47, 25, 43, 35, 25, 22, 10, 33, 8, 9].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “Contrastive Learning techniques use a contrastive loss [28] to learn representations either by predicting parts of the data [32, 33, 61], or discriminating between individual training instances [17, 19, 31, 34, 53, 86, 88, 94].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Interestingly, most of the pretext tasks can be integrated into a contrastive learning framework [20, 4, 47, 17, 30], which maintains relative consistency between the representations of an image and its augmented view.”
Decoupled Contrastive Learning (2021, background): “spread [32] makes use of data augmentation invariant and the spread-out property of instance to learn features.”
Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning (2020, background): “This line of investigation can be traced back to [14], and after years of progress [34, 29, 21, 38, 1, 35], transfer performance superior to supervised methods was achieved by MoCo [18] on a broad range of downstream tasks.”