A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics

By Yunchao Gong, Qifa Ke, Michael Isard, Svetlana Lazebnik. 2012

588 citations recorded by the source corpus.

This paper investigates the problem of modeling Internet images and associated text or tags for tasks such as image-to-image search, tag-to-image search, and image-to-tag search (image annotation). We start with canonical correlation analysis (CCA), a popular and successful approach for mapping visual and textual features to the same latent space, and incorporate a third view capturing high-level image semantics, represented either by a single category or multiple non-mutually-exclusive concepts. We present two ways to train the three-view embedding: supervised, with the third view coming from ground-truth labels or search keywords; and unsupervised, with semantic themes automatically obtained by clustering the tags. To ensure high accuracy for retrieval tasks while keeping the learning process scalable, we combine multiple strong visual features and use explicit nonlinear kernel mappings to efficiently approximate kernel CCA. To perform retrieval, we use a specially designed similarity function in the embedded space, which substantially outperforms the Euclidean distance. The resulting system produces compelling qualitative results and outperforms a number of two-view baselines on retrieval tasks on three large-scale Internet image datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “Other approaches use user-deﬁned keywords or tags [20, 28, 43, 19] or algorithmically-generated topics [1, 31] to the same end.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “e search across modalities [36]. This idea is at the core of our versatile networks. With larger datasets [47, 69, 71, 92, 96], many works have proﬁted from learning such a joint visual-textual space [16, 18, 28, 29, 40, 52, 56, 62, 68, 84, 85, 87, 88, 93]. Recently, instructional videos became a po”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “A number of image-text embedding approaches has been developed based on Canonical Correlation Analysis (CCA) [12, 16, 47, 57].”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “Methods such as those presented in [29, 13, 27, 22] use Canonical Correlation Analysis (CCA) for establishing relationships between data of different modalities.”
Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Beyond CCA: Moment Matching for Multi-View Models (2016, neutral): “, 2002) or images aligned with text data (e.g., Hardoon et al., 2004; Gong et al., 2014).”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “This merging resulting in a reduction of the number of bounding boxes in the dataset by 19.8% and 5.9% fewer coreference chains.”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “[26] have shown that properly normalized CCA [14] on top of state-of-the-art image and text features can outperform considerably more complex models.”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, methodology): “This method, nCCA, provides high-quality retrieval results, improving over the original CCA performance signiﬁcantly [14].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “One way to leverage language in vision is to ﬁnd a joint embedding space for both visual and textual modalities [7], [8], [9], [19], [20], [21], [22], [23].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, methodology): “[2] showed that properly normalized CCA [23] with stateof-the-art image and text features can outperform much more complicated models.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, methodology): “Canonical Correlation Analysis (CCA) was used in several previous works for learning joint embeddings for the retrieval task [10, 13, 29, 38].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “earn the task for the target. These methods are related, on one hand, to the feature transformation based homogeneous DA methods described in Section 3.1 and, on another hand, to multi-view embedding [93, 110, 99, 111, 112, 113], where different views are embedded in a common latent space. Therefore”
Exploring the Challenges towards Lifelong Fact Learning (2018, methodology): “We chose this architecture due its superior performance compared to other joint-embedding models like [9,13,24] and its competitive performance to multi-class cross-entropy.”