Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search

By Yue Gao, Meng Wang, Zhengjun Zha, Jialie Shen, Xuelong Li, Xindong Wu. IEEE Transactions on Image Processing, 2013

463 citations recorded by the source corpus.

Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/866cc0eafc25c7216d8f242b2d3d401719585b5e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Rank Using User Clicks and Visual Features for Image Retrieval (2015, background): “tures are integrated in terms of a linear model, and the visual features are considered in terms of a hypergraph regularizer [22], [33], [34], [36]–[38], [51] which captures high-order relationships in building the graph.”
Multi-view hypergraph learning by patch alignment framework (2013, background): “DLA is supervised and could also be extended to a semisupervised approach.”
Exploring Principles-of-Art Features For Image Emotion Recognition (2014, methodology): “In addition, we will consider using social network (e.g., Flickr) data, combining the descriptions and images to jointly learn the expected emotion of speciﬁed image based on visual-textual-social features [9] and analyzing the comments to distinguish expected emotion and actual emotion.”
Global Contrast Based Salient Region Detection (2015, background): “adaptive compression of images [6], content-aware image editing [7], [8], image retrieval [9]–[11], etc.”
Physiological Signals-based Emotion Recognition via High-order Correlation Learning (2019, background): “Reference[12] proposed a hypergraph to model the relationship of user-tagged images with respect to both visual and textual information.”
Object Detection With Deep Learning: A Review (2018, background): “Besides, given a specific application, the information from different modalities, such as text [212], thermal data [205] and images [65], can be fused together to achieve a more discriminant network.”
Unifying Virtual and Physical Worlds (2017, methodology): “…classiﬁed into two categories: early fusion [Blaschko and Lampert 2008; Chaudhuri et al. 2009] and joint learning [Bruno and Marchand-Maillet 2009; Gao et al. 2013a; Kumar and Daum´e 2011; Kumar et al. 2011; Liu et al. 2012; Zhang et al. 2015; Zeng et al. 2006; Boden et al. 2012; Cai et al.…”
Block-Row Sparse Multiview Multilabel Learning for Image Classification (2016, background): “Literature has shown that conducting image classification with multiview learning can further improve the classification performance [6], [8], [23]–[25].”
Efficient kNN classification algorithm for big data (2016, methodology): “Thus, in order to show the effectiveness of LC-kNN algorithm, we took the kNN as the baseline and made a comparison between kNN, LC-kNN and RC-kNN (random clustering kNN), and several real datasets from UCI [2], medical imaging datasets [6,7], and LIBSVM [3] datasets.”
Classifying tag relevance with relevant positive and negative examples (2013, background): “We propose a system that selects positive and negative examples, deemed most relevant with respect to the given tag from crowd-annotated images.”
Hypergraph Neural Networks (2018, background): “In (Gao et al. 2013), a l 2 regularize on the weights is introduced to learn optimal hyperedge weights.”
DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection (2016, background): “In recent years, researchers have developed many computational models for salient object detection and applied them to benefit many other applications, such as image summarization [1], segmentation [2], retrieval [3], and editing [4].”