DeViSE: A Deep Visual-Semantic Embedding Model

By Andrea Frome, Greg S. Corrado, Jonathon Shlens, Samy Bengio, Joseph Dean, Marc'Aurelio Ranzato, Tomas Mikolov. 2013

3,053 citations recorded by the source corpus.

Subfields: Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Devi Parikh
2. Zhe Gan
3. Mohit Bansal
4. Trevor Darrell
5. Marcus Rohrbach
6. Jianfeng Gao
7. Qi Wu
8. Dhruv Batra

Citations in the corpus

• Distributed Representations of Sentences and Documents (2014, methodology): “Representing phrases is a recent trend and received much attention (Mitchell & Lapata, 2010; Zanzotto et al., 2010; Yessenalina & Cardie, 2011; Grefenstette et al., 2013; Mikolov et al., 2013c).”
• ImageNet Large Scale Visual Recognition Challenge (2014, background): “Additionally, several influential lines of research have emerged, such as large-scale weakly supervised localization work of (Kuettel et al., 2012) which was awarded the best paper award in ECCV 2012 and large-scale zero-shot learning, e.g., (Frome et al., 2013).”
• Learning Transferable Visual Models From Natural Language Supervision (2021, background): “Over time work explored many combinations of training objective, transfer, and more expressive models and steadily improved performance (Frome et al., 2013; Socher et al., 2014; Karpathy et al., 2014; Kiros et al., 2014; Faghri et al., 2017).”
• Self-Supervised MultiModal Versatile Networks (2020, background): “WSABIE [86] and DeVise [24] introduced the idea of embedding text and image in the same space.”
• Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, positive): “on analysis, and achieve good results for annotation and segmentation. Similar ideas are employed for image captioning [14] and text-based image retrieval [30]. In particular, the seminal work DeViSE [8] proposes to identify visual objects using semantic information gleaned from un-annotated text. T”
• Shaping Visual Representations with Language for Few-Shot Classification (2019, neutral): “Language has been shown to assist visual classification in various settings, including traditional visual classification with no transfer [16] and with language available at test time in the form of class labels or descriptions for zero-[10, 11, 27] or few-shot [24, 33] learning.”
• Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “nticEmbedding: Joint visual-semantic models have shown excellent performance on several multimedia tasks, e.g., cross-modal retrieval [18, 31, 38, 55], image captioning [23, 36], image classification [10, 12, 20] video summarization [4, 41]. Crossmodal retrieval methods require computing semantic si”
• Learning with Latent Language (2017, methodology): “ve been used in conjunction with training examples to guide the discovery of logical descriptions of concepts (Ling et al., 2017; Srivastava et al., 2017), and used as an auxiliary loss for training (Frome et al., 2013), analogously to the Meta+Joint baseline in this paper. Structured language-like ”
• TensorFlow: A system for large-scale machine learning (2016, neutral): “In recent years, machine learning has driven advances in many different fields [3, 5, 23, 24, 30, 27, 40, 45, 48, 50, 55, 68, 69, 73, 76].”
• Learning Visual Features from Large Weakly Supervised Data (2015, background): “Frome et al. [14] show that convolutional networks trained jointly on annotated image data and a large corpus of unannotated texts can be used for zero-shot learning.”
• Predicting Deep Zero-Shot Convolutional Neural Networks Using Textual Descriptions (2015, methodology): “In [9], label features are produced by an unsupervised skip-gram model [18] trained on Wikipedia articles, while the image features are produced by a CNN trained on Imagenet [14].”
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “The deep visual semantic embedding model [5] was proposed as a way of performing zero-shot object recognition and was used as a baseline by [15].”
• Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Related work has also been presented at NIPS (Socher et al., 2013b; Frome et al., 2013).”
• Conditional Generative Adversarial Nets (2014, background): “Works such as [3] have shown that even a simple linear mapping from image feature-space to word-representation-space can yield improved classification performance.”
• SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “DeViSE [19] jointly embeds images and textual class labels within a shared semantic space, allowing the model to recognize classes that were not explicitly trained for.”
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “We and others at Google have performed a wide variety of research using DistBelief including work on unsupervised learning [31], language representation [35, 52], models for image classification and object detection [16, 48], video classification [27], speech recognition [56, 21, 20],”