Selfie: Self-supervised Pretraining for Image Embedding

By Trieu H. Trinh, Minh-Thang Luong, Quoc V. Le. 2019

117 citations recorded by the source corpus.

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-out patches in an input image, our method learns to select the correct patch, among other "distractor" patches sampled from the same image, to fill in the masked location. This classification objective sidesteps the need for predicting exact pixel values of the target patches. The pretraining architecture of Selfie includes a network of convolutional blocks to process patches followed by an attention pooling network to summarize the content of unmasked patches before predicting masked ones. During finetuning, we reuse the convolutional weights found by pretraining. We evaluate Selfie on three benchmarks (CIFAR-10, ImageNet 32 x 32, and ImageNet 224 x 224) with varying amounts of labeled data, from 5% to 100% of the training sets. Our pretraining method provides consistent improvements to ResNet-50 across all settings compared to the standard supervised training of the same network. Notably, on ImageNet 224 x 224 with 60 examples per class (5%), our method improves the mean accuracy of ResNet-50 from 35.6% to 46.7%, an improvement of 11.1 points in absolute accuracy. Our pretraining method also improves ResNet-50 training stability, especially on low data regime, by significantly lowering the standard deviation of test accuracies across different runs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “, 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “Other approaches use contrastive losses for context prediction [20, 23], mutual information maximization [21, 53, 54], predicting masked regions [55], and clustering [56].”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, methodology): “Grid representation have been widely used in vision tasks, including self-supervised learning (Oord et al., 2018; Henaff et al., 2019; Trinh et al., 2019; Gidaris et al., 2020; Noroozi and Favaro, 2016) and image generation (van den Oord et al., 2017; Lin et al., 2019).”
A review on weight initialization strategies for neural networks (2021, methodology): “Trinh et al. (2019) have presented a self-supervised pre-training approach for initialization of Resnet-50 CNN architecture.”
When Does Self-supervision Improve Few-shot Learning? (2019, background): “…Joulin, 2017), clusters (Caron et al., 2018; 2019; Wu et al., 2018), number of objects (Noroozi et al., 2017), missing patches (Pathak et al., 2016; Trinh et al., 2019), motion segmentation labels (Pathak et al., 2017), artifacts (Jenni & Favaro, 2018), next patch feature (Oord et al., 2018;…”
Rethinking the Value of Labels for Improving Class-Imbalanced Learning (2020, background): “[49] Trieu H Trinh, Minh-Thang Luong, and Quoc V Le.”
Learning Representations by Predicting Bags of Visual Words (2020, background): “, ResNet-50) with extra network components, such as multiple stacks of attention modules as in [61] or PixelCNN-like autoregressors as in [46], that can make the overall architecture computationally intensive.”
OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning (2020, background): “In that respect, it is closer to contrastive-based approaches that target to learn context-aware representations by predicting (in a contrastive way) the state of missing image patches [35, 49, 65].”
UNITER: UNiversal Image-TExt Representation Learning (2019, methodology): “Self-supervised learning utilizes original data as its own source of supervision, which has been applied to many Computer Vision tasks, such as image colorization [59], solving jigsaw puzzles [31,48], inpainting [35], rotation prediction [15], and relative location prediction [10].”
SimMIM: a Simple Framework for Masked Image Modeling (2021, methodology): “CPC [21,45] predicts patches via a verification task in each batch with a contrastive predictive coding loss.”
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, methodology): “The MLM strategy has also inspired a lot of pre-training tasks [2, 7, 18, 25, 46].”
Generative Pretraining From Pixels (2020, result): “…Predictive Coding (Oord et al., 2018) which also utilizes a autoregressive prediction objective, but in a learned latent space, and to Selﬁe (Trinh et al., 2019) Several of our ﬁndings are consistent with their results, including the beneﬁts of scale and the non-monotonic performance of…”
Towards Domain-Agnostic Contrastive Learning (2020, background): “(b) another class of methods constructs positive and negative samples by defining the local and global context in a sample (Hjelm et al., 2019; Sun et al., 2020; Veličković et al., 2019; Bachman et al., 2019; Trinh et al., 2019).”
Graph Contrastive Learning with Augmentations (2020, background): “Self-supervision with handcrafted pretext tasks [29, 30, 31, 32] relies on heuristics to design, and thus could limit the generality of the learned representations.”
TabNet: Attentive Interpretable Tabular Learning (2019, background): “Recent work for text (Devlin et al. 2018) and image (Trinh, Luong, and Le 2019) data has shown significant advances – driven by the judicious choice of the unsupervised learning objective (masked input prediction) and attention-based deep learning.”
Vision Transformers are Robust Learners (2021, methodology): “Although Transformers have been used in prior works [5, 6] it was only until 2020, the performance of Transformers were on par with the SOTA CNNs on standard image recognition tasks [7, 8, 1].”