Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces

By Lluis Gomez i Bigorda, Yash J. Patel, Marçal Rusiñol, Dimosthenis Karatzas, C. V. Jawahar. 2017

130 citations recorded by the source corpus.

End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multi-modal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or natural-supervised approaches.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Zhang et al. (2020), Gomez et al. (2017), Joulin et al. (2016), and Desai & Johnson (2020) all introduce methods which learn visual representations from text paired with images but describe their approaches as unsupervised, self-supervised, weakly supervised, and supervised respectively.”
Learning Visual Representations with Caption Annotations (2020, methodology): “(ii) The proposed TPCluster outperforms the current state of the art for training with captions, TPLDA [20], for all three datasets.”
Learning to Prompt for Vision-Language Models (2021, background): “, 2013), multi-label classification (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al.”
The Challenges of Continuous Self-Supervised Learning (2022, neutral): “proposed from generative models such as denoising auto-encoders [ 77 ], sparse coding [ 36 , 54 , 55 ], inpainting [ 59 ] and colorization [ 18 , 34 , 86 ], to methods that learn representations predictive of spatial context [ 19 , 22 , 53 ], temporal context [ 21 , 47 , 58 , 60 , 78 , 79 ], or conc”
Improvements to Context Based Self-Supervised Learning (2017, neutral): “1 – – Gomez [15] 55.”
Conditional Prompt Learning for Vision-Language Models (2022, background): “In terms of cross-modality alignment, common approaches include metric learning [12], multi-label classification [16, 26], and n-gram language learning [31].”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “Language supervision Joulin et al. (2016); Gomez et al. (2017); Zhang et al. (2020); Sariyildiz et al. (2020); Desai & Johnson (2021) demonstrate the effectiveness of learning transferable visual features from language supervision.”
Scene Text Visual Question Answering (2019, background): “On the other hand, they comprise words obtained by exploiting the semantic understanding of the scene, in the form of the output of a dynamic lexicon generation model [43, 14].”
Multi-task Self-Supervised Learning for Human Activity Detection (2019, background): “Self-supervision has been actively studied recently in the vision domain, and several surrogate tasks have been proposed for learning representations from static images, videos, sound, and in robotics [3, 13, 15, 18, 19, 24, 30, 34, 42, 47, 51, 52, 54, 74].”
Unsupervised Pre-Training of Image Features on Non-Curated Data (2019, unknown): “Some methods [9, 17, 32] aim at learning visual features from non-curated data streams.”
Self-Supervised Learning from Web Data for Multimodal Retrieval (2019, background): “Gomez, Patel et al. [14, 15] use LDA [16] to extract topic probabilities from a bunch of Wikipedia articles and train a CNN to embed their associated images in the same topic space.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Table III summarizes different SSL losses. b) Images and text: In the study conducted by Gomez et al. [148], the authors employed a topic modeling framework to project the text of an article into the topic probability space.”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Some studies [12, 16, 17, 44, 47, 53] pro-pose to learn from image-caption pairs.”
Exploring Visual Engagement Signals for Representation Learning (2021, methodology): “Other studies convert language to discrete labels or continuous probability distributions, such as individual word, part-of-speech (POS) tags, clustering assignments of sentence features, and topic modeling [38, 46, 19, 2].”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, background): “from the image). Finally, we describe prior work on understanding image advertisements and visual persuasion. Visual-semantic embeddings and image captioning. Work in learning joint image-text spaces [3,5,10,12,20,36] aims to project an image and a matching piece of text in such a way that the dista”
Extract Free Dense Labels from CLIP (2021, background): “Some studies [44,16,17,48,12,53] propose to learn from image-caption pairs.”
Multimodal Contrastive Training for Visual Representation Learning (2021, background): “Recently, several works [46, 35, 18, 19, 10, 49] propose pre-training approaches that use semantically dense captions to learn visual representations from scratch.”
Mining on Manifolds: Metric Learning Without Labels (2018, background): “Finally, other cases utilize different modalities, such as learning visual descriptors from text information [14], or learning audio descriptors from unlabeled videos [2].”
Infrared-Visible Cross-Modal Person Re-Identification with an X Modality (2020, methodology): “…to fully-supervised algorithms, self-supervised learning has provided considerable improvements in cross-modal learning, like image and context cross-modal learning (Doersch, Gupta, and Efros 2015; Gomez et al. 2017), instructions and trajectories cross-modal learning (Wang et al. 2019b), etc .”