Large scale image annotation: learning to rank with joint word-image embeddings

By Jason Aaron Edward Weston, Samy Bengio, Nicolas Usunier. 2010

461 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “While initial efforts focused primarily on predictive objectives over time research shifted towards learning joint multi-modal embedding spaces with techniques like kernel Canonical Correlation Analysis and various ranking objectives (Weston et al., 2010; Socher & Fei-Fei, 2010; Hodosh et al., 2013)”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, background): “Popular approaches include learning joint image-word embeddings [4, 5] as well as embedding images and sentences into a common space [6, 15].”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “…of Farhadi et al. (2010), who incorporate Lin's similarity into their model, but do not evaluate what benefit they obtain from it, WordNet's hypernym-hyponym relations have only been used superficially for associating images and text (Weston et al., 2010; Ordonez et al., 2011; Gupta et al., 2012).”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “Multi-modal feature learning Learning a joint and aligned feature space for vision-and-language has been a long standing problem in artiﬁcial intelligence [40,48].”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Similar to CLIP and ALIGN, our work also learns the mapping between images and texts, which is related to many previous works, such as (Hironobu et al., 1999; Weston et al., 2010; Socher and Fei-Fei, 2010; Socher et al., 2013; Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et a”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Learning from multimodal data such as text, images, and audio is a longstanding research challenge in machine learning [31, 51, 56, 83, 86].”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, methodology): “To directly test how much of the CBT can be resolved by good quality dense representations of words (word embeddings), we implement a supervised embedding model similar to that of (Weston et al., 2010).”
Representation Learning: A Review and New Perspectives (2012, background): “Another family of approaches, more in the spirit of Contrastive Divergence, relies on distinguishing positive examples (of the training distribution) and negative examples obtained by perturbations of the positive examples (Collobert and Weston, 2008; Bordes et al., 2012; Weston et al., 2010).”
Large-scale Simple Question Answering with Memory Networks (2015, methodology): “We use the WARP loss (Weston et al., 2010) to speed up training, and Adagrad (Duchi et al., 2011) as SGD algorithm multi-threaded with HogWild!”
Towards Universal Paraphrastic Sentence Embeddings (2015, background): “…learning formulation draws from a large body of related work on learning input representations in order to maximize similarity in the learned space (Weston et al., 2010; Yih et al., 2011; Huang et al., 2013; Hermann & Blunsom, 2014; Socher et al., 2014; Faruqui & Dyer, 2014; Bordes et al.,…”
Question Answering with Subgraph Embeddings (2014, methodology): “As in (Weston et al., 2010), we train our model using a margin-based ranking loss function.”
Learning Deep Representations of Fine-Grained Visual Descriptions (2016, background): “Embedding labels in an euclidean space is an effective way to model latent relationships between classes [4, 47].”
Deep Learning: Methods and Applications (2014, methodology): “The earlier WSABIE system as described in [388, 389] adopted a shallow architecture and trained a joint embedding model of both images and labels.”
Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction (2013, methodology): “See (Weston et al., 2010; Bordes et al., 2013) for details.”
A semantic matching energy function for learning with multi-relational data (2013, methodology): “To train the parameters of the energy function E we loop over all of the training data resources and use stochastic gradient descent with a ranking objective in spired by [7].”
Learning Fine-Grained Image Similarity with Deep Ranking (2014, background): “More importantly, it also includes features learn d through image annotation data, such as features or embeddings developed in [24].”
Zero-Shot Learning — The Good, the Bad and the Ugly (2017, methodology): “Following the heuristic in [18], [3] selects αi = 1/i which puts high emphasis on the top of the rank list.”
Open Question Answering with Weakly Supervised Embedding Models (2014, methodology): “Our scoring approach is inspired by previous work for labeling images with words [23], which we adapted, replacing images and labels by questions and triples.”