Learning Visual Representations using Images with Captions

By Ariadna Quattoni, Michael Collins, Trevor Darrell. 2007

133 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Quattoni et al. (2007) demonstrated it was possible to learn more data efficient image representations via manifold learning in the weight space of classifiers trained to predict words in captions associated with images.”
Learning Visual Representations with Caption Annotations (2020, methodology): “esentations from scratch for purely visual tasks by leveraging captions. Learning visual features using text. Only few works have taken advantage of the companion text to learn image representations. [49] creates and solves auxiliary prediction tasks from images with associated captions. [37] constr”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Early work explored learning visual representations from image captions, even before the advent of deep learning [44].”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Learning from multimodal data such as text, images, and audio is a longstanding research challenge in machine learning [31, 51, 56, 83, 86].”
A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics (2012, methodology): “The last two lines of Table 4 report baseline comparisons to structural learning (Ando and Zhang, 2005; Quattoni et al., 2007) and Wsabie (Weston et al.”
Transfer learning for image classification with sparse prototype representations (2008, methodology): “linear subspaces) using data from related tasks and can also be used for transfer learning on this data [14].”
Zero-Shot Reward Specification via Grounded Natural Language (2022, background): “A steady series of innovations have advanced grounded language modeling, from early work on multimodal translation and fusion models (Barnard et al., 2003; Quattoni et al., 2007; Guadarrama et al., 2016), to large-scale joint embedding models (Frome et al.”
Trace Norm Regularization: Reformulations, Algorithms, and Multi-Task Learning (2010, methodology): “Multi-task learning, gene expression pattern analysis, trace norm regularization, convex optimiza- tion, duality, semidefinite programming, proximal gradient method.”
Learning to Recognize Objects from Unseen Modalities (2010, background): “In [14] a transfer learning approach was proposed to learn a discriminatively trained latent space over images and their captions.”
Multimodal semi-supervised learning for image classification (2010, methodology): “Others have used image captions to learn face recognition models without manual supervision [2], or to learn low dimensional image representations by predicting caption words and can be transferred to other image classification problems [17].”
A convex formulation for learning shared structures from multiple tasks (2009, methodology): “Multi-task learning has been investigated by many researchers from di.erent perspectives such as sharing hidden units of neural networks among similar tasks (Caruana, 1997; Baxter, 2000), modeling task relatedness using the common prior distribution in hierarchical Bayesian models (Bakker…”
A new approach to cross-modal multimedia retrieval (2010, background): “Lastly, it is possible to design multimodal systems by using text annotations to construct a semantic space [25].”
Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections (2014, background): “to multi-view learning [13, 7], transfer learning [14–16], and methods that use Internet data to help recognition [17–21].”
Does language help generalization in vision models? (2021, background): “Indeed, it has been shown that vision models have better generalization capabilities when coupled with language [18, 21].”
Cross-Language Text Classification Using Structural Correspondence Learning (2010, methodology): “Following Ando and Zhang (2005a) and Quattoni et al. (2007) we choose w for the target task to be w∗ = θT v∗, where v∗ is defined as follows: v∗ = argmin v∈Rk ∑ (x,y)∈DS L(y, (θTv)Tx) + λ 2 ‖v‖2 (3) Since (θT v)T = vT θ it follows that this view of CL-SCL corresponds to the induction of a new…”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “Some studies [12, 16, 17, 44, 47, 53] pro-pose to learn from image-caption pairs.”
Clustered Multi-Task Learning Via Alternating Structure Optimization (2011, background): “shown to be effective in many practical applications [2, 11, 12].”