Learning Representations by Predicting Bags of Visual Words

By Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, Matthieu Cord. 2020

130 citations recorded by the source corpus.

Self-supervised representation learning targets to learn convnet-based image representations from unlabeled data. Inspired by the success of NLP methods in this area, in this work we propose a self-supervised approach based on spatially dense image descriptions that encode discrete visual concepts, here called visual words. To build such discrete representations, we quantize the feature maps of a first pre-trained self-supervised convnet, over a k-means based vocabulary. Then, as a self-supervised task, we train another convnet to predict the histogram of visual words of an image (i.e., its Bag-of-Words representation) given as input a perturbed version of that image. The proposed task forces the convnet to learn perturbation-invariant and context-aware image features, useful for downstream image understanding tasks. We extensively evaluate our method and demonstrate very strong empirical results, e.g., our pre-trained self-supervised representations transfer better on detection task and similarly on classification over classes ``unseen'' during pre-training, when compared to the supervised case. This also shows that the process of image discretization into visual words can provide the basis for very powerful self-supervised approaches in the image domain, thus allowing further connections to be made to related methods from the NLP domain that have been extremely successful so far.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, background): “Several other works echo this direction, showing that one can match more elaborate representations [26, 27], train features matching them to a uniform distribution [6] or by using whitening [23, 81].”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “In unsupervised and self-supervised approaches [9, 10, 22, 25, 34, 56] the model is trained on unlabeled data.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Our work is also related to clustering-based meth-ods [2, 4, 7, 8, 19, 29, 58, 61, 62, 67].”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “In OBoW Gidaris et al. (2020) and Gidaris et al. (2021) the dictionary is obtained by off-line or on-line clustering.”
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, positive): “Grid representation have been widely used in vision tasks, including self-supervised learning (Oord et al., 2018; Henaff et al., 2019; Trinh et al., 2019; Gidaris et al., 2020; Noroozi and Favaro, 2016) and image generation (van den Oord et al., 2017; Lin et al., 2019).”
MARE: Self-Supervised Multi-Attention REsu-Net for Semantic Segmentation in Remote Sensing (2021, methodology): “Other approaches fulfill the goal of reconstructing a target image, after perturbing it [16,17].”
Visual Prompting via Image Inpainting (2022, background): “Recent work has shown that self-supervised pretraining can generate powerful representations for transfer learning, even outperforming its supervised counterparts on challenging vision benchmarks [37, 18, 9, 21, 6, 10, 35, 16].”
Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning (2020, methodology): “In self-supervised learning, BowNet [26] trains a student to match the BoW representations produced by a self-supervised pre-trained teacher.”
Contrastive Learning With Stronger Augmentations (2021, methodology): “Also, BoWNet [53] and OBOW [54] even included CutMix [55] as stronger augmentation to contribute”
Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning (2022, methodology): “Besides, BoWNet [35] and OBoW [36] also use the vanilla CutMix by mixing two randomly sampled images.”
Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “Most related to our work are regressionbased methods that instead try to directly regress a representation extracted from a different view of the image [27, 68].”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Deep clustering models learn discriminative representations by grouping similar instances from the same cluster together [131], [132], [133], [134], [146], [147], [148], [149], [150], [151].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “ from unlabelled images. Some tasks leverage the spatial context in images [24, 74] to train CNNs, while others create pseudo classiﬁcation labels via artiﬁcial rotations [33], or clustering features [8, 14, 15, 34]. Colorization [110, 111], inpainting [81], solving jigsaw puzzles [75], as well as t”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “V-JEPA obtains competitive performance using a finetuning protocol.”
Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “80], instance discrimination [28,27], image discretization [34,35], masked image modeling [59,31,74], and information maximization [30,97,5].”