Graph-RISE: Graph-Regularized Image Semantic Embedding

By Da-Cheng Juan, Chun-Ta Lu, Z. Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, Sujith Ravi. 2019

43 citations recorded by the source corpus.

Learning image representations to capture fine-grained semantics has been a challenging and important task enabling many applications such as image search and clustering. In this paper, we present Graph-Regularized Image Semantic Embedding (Graph-RISE), a large-scale neural graph learning framework that allows us to train embeddings to discriminate an unprecedented O(40M) ultra-fine-grained semantic labels. Graph-RISE outperforms state-of-the-art image embedding algorithms on several evaluation tasks, including image classification and triplet ranking. We provide case studies to demonstrate that, qualitatively, image retrieval based on Graph-RISE effectively captures semantics and, compared to the state-of-the-art, differentiates nuances at levels that are closer to human-perception.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Graph Learning

Quoc V. Le

Citations in the corpus

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “We select the top 16 box proposals and featurize each of them with Graph-RISE [34, 35], similar to [19].”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “High-quality visual representations for classiﬁcation or retrieval are usually pre-trained on large-scale labeled datasets (Mahajan et al., 2018; Kolesnikov et al., 2020; Dosovitskiy et al., 2021; Juan et al., 2020).”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, methodology): “While the text input to the model was a bag of words, the image tower, the image was first embedded in a manner similar to [16].”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “On the flip side, the query-based training of the Graph-RISE image embedding models (Juan et al., 2019) is a powerful demonstration of the power of words to produce stronger image representations.”
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (2019, methodology): “In particular, we (i) leverage ultra-ﬁne-grained semantic labels (e.g., “golden gate bridge” vs. “bridge”) for featurization (Juan et al., 2019); and, (ii) focus on scenarios in which object detection modules trained on Visual Genome (VG) (Krishna et al., 2017) are applied to out-of-domain images:…”