Beyond Instance-Level Image Retrieval: Leveraging Captions to Learn a Global Visual Representation for Semantic Retrieval

By Albert Gordo, Diane Larlus. 2017

97 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Learning Visual Representations with Caption Annotations (2020, background): “ms, and trains CNNs by predicting these labels. [20] extracts topic models for Wikipedia pages using latent Dirichlet allocation and trains a CNN to embed their associated images in this topic space. [23] uses captions to learn image representations for the specic task of semantic retrieval. We argu”
Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “[13] use METEOR scores between captions as a similarity metric for specializing embeddings for image retrieval, but do not directly [26], we propose few-shot classiﬁcation models whose learned representations are constrained to predict natural language descriptions of the task during training, in…”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “A similar idea is explored in the work of Gordo and Larlus [14] in these same proceedings, where image cap-tions are leveraged to learn a global visual representation for semantic retrieval.”
Unified Contrastive Learning in Image-Text-Label Space (2022, neutral): “(i) Inspired by the success of BERT [13], the first line of research focuses on learning generic multi-modal fusion layers based on masked token prediction and/or image-text matching, given the pre-extracted features from visual and textual encoder [17, 31, 38, 39, 44, 51, 63, 73].”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Some studies [12, 16, 17, 44, 47, 53] pro-pose to learn from image-caption pairs.”
Semantic bottleneck for computer vision tasks (2018, methodology): “esentations that are not intelligible at all. In contrast, image captioning [8,9] is by nature producing intelligible representations and can be used to index images. As an illustration, Gordo et al. [10] addressed the task of retrieving images that share the same semantics as the query image using ”
Learnable PINs: Cross-Modal Embeddings for Person Identity (2018, background): “While joint embeddings have been researched intensively for images and text, [4, 13, 14, 22, 41], they have also started to gain traction for audio and vision [3, 31, 37].”
Extract Free Dense Labels from CLIP (2021, background): “Some studies [44,16,17,48,12,53] propose to learn from image-caption pairs.”
Exploring Hate Speech Detection in Multimodal Publications (2019, methodology): “This approach is applied in tasks such as image captioning [14, 13] and multimodal image retrieval [9, 8].”
Large-Scale Visual Relationship Understanding (2018, result): “ing that although theoretically traditional triplet loss can pushes the margin as much as possible when m = 1, most previous works (e.g., Kiros, Salakhutdinov, and Zemel (2014); Faghri et al. (2018); Gordo and Larlus (2017)) adopted a small mto allow slackness during training. It is also unclear how”
Self-Supervised Learning from Web Data for Multimodal Retrieval (2019, background): “Gordo & Larlus [13] use captions associated to images to learn a common embedding space for images and text through which they perform semantic image retrieval.”
Multimodal Contrastive Training for Visual Representation Learning (2021, background): “Recently, several works [46, 35, 18, 19, 10, 49] propose pre-training approaches that use semantically dense captions to learn visual representations from scratch.”