Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora

By Richard Socher, Li Fei-Fei. 2010

257 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “…initial efforts focused primarily on predictive objectives over time research shifted towards learning joint multi-modal embedding spaces with techniques like kernel Canonical Correlation Analysis and various ranking objectives (Weston et al., 2010; Socher & Fei-Fei, 2010; Hodosh et al., 2013).”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “tions about image labels not observed during training, and improved such zero-shot predictions dramatically across thousands of novel labels never seen by the visual model. The idea has been extended [34, 16,25], showing that leveraging pre-trained linguistic information is highly effective to align”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “to a common space based on extracting a triplet of object, action, and scene is presented. A number of image-text embedding approaches has been developed based on Canonical Correlation Analysis (CCA) [12, 16, 47, 57]. Ranking loss has been used for training the embedding in most recent works relatin”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Socher and FeiFei [13] and Hodosh et al.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “RNN models (Pollack, 1990; Goller and K¨uchler, 1996; Socher et al., 2011b; Socher et al., 2011a) provided a novel way of combining word vectors for longer phrases that moved beyond simple averaging.”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “…data sets, but does not cause any problems here, since our training data consists of only 6,000 items ( KCCA has been successfully used to associate images (Hardoon et al., 2004; Hwang & Grauman, 2012; Hardoon et al., 2006) or image regions (Socher & Li, 2010) with individual words or sets of tags.”
Zero-Shot Learning Through Cross-Modal Transfer (2013, methodology): “[31] project words and image regions into a common space using kernelized canonical correlation analysis to obtain state of the art performance in annotation and segmentation.”
Beyond CCA: Moment Matching for Multi-View Models (2016, methodology): “Examples of applications and extensions of Gaussian CCA are the works by Socher & Fei-Fei (2010), for mapping visual and textual features to the same latent space, and Haghighi et al. (2008), for machine translation applications.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Early methods [19,31,37,65,79] focus on filling templates with extracted objects, attributes, and relationships.”
Attention on Attention for Image Captioning (2019, methodology): “Earlier approaches to image captioning are rule/template-based [48, 34] which generate slotted caption templates and use the outputs of object detection [30, 39, 38], attribute prediction and scene recognition to fill in the slots.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “During training, the input to our model is a set of images and their corresponding sentence descriptions (Figure 2).”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, background): “Several pioneering approaches have explored the task of describing images with natural language [1, 27, 12, 34, 42, 43, 28, 20].”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, methodology): “While early approaches were based on caption templates filled by using the output of pre-trained object detectors [14], [15], almost all recent captioning models integrate recurrent neural networks as language models [6], 2020 IEEE International Conference on Robotics and Automation (ICRA) 31 May - ”
Deep Learning: Methods and Applications (2014, methodology): “For example, Socher and Fei-Fei [341] project words and images into the same space using kernelized canonical correlation analysis.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “(2) Canonical Correlation Analysis (CCA) [24]: CCA is to build a video-language joint space and generate the SVO triplet by k-nearest-neighbors search in the sentence pool.”
Jointly Modeling Deep Video and Compositional Text to Bridge Vision and Language in a Unified Framework (2015, methodology): “CCA CCA (Socher and Fei-Fei 2010) has been used to build an image-text space, we use CCA as a baseline to build the video-language space and compare video retrieval/text retrieval with our method.”
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation (2014, background): “Other works [49, 9, 42] have focused on developing bi-directional mappings.”