Ultra Fine-Grained Image Semantic Embedding
By Da-Cheng Juan, Chun-Ta Lu, Zhuguo Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, Sujith Ravi. 2020
23 citations recorded by the source corpus.
"How to learn image embeddings that capture fine-grained semantics based on the instance of an image?" "Is it possible for such embeddings to further understand image semantics closer to humans' perception?" In this paper, we present, Graph-Regularized Image Semantic Embedding (Graph-RISE), a web-scale neural graph learning framework deployed at Google, which allows us to train image embeddings to discriminate an unprecedented O(40M) ultra-fine-grained semantic labels. The proposed Graph-RISE outperforms state-of-the-art image embedding algorithms on several evaluation tasks, including kNN search and triplet ranking: the accuracy is improved by approximately 2X on the ImageNet dataset and by more than 5X on the iNaturalist dataset. Qualitatively, image retrieval from one billion images based on the proposed Graph-RISE effectively captures semantics and, compared to the state-of-the-art, differentiates nuances at levels that are closer to human-perception.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jason Baldridge
- Yinfei Yang
- Mohit Bansal
- Eugene Ie
- Alexander Ku
- Daniel Matthew Cer
- Soravit Changpinyo
- Nan Ding
In Computer Vision and Pattern Recognition
- Jason Baldridge
- Luc Van Gool
- Cordelia Schmid
- Trevor Darrell
- Dacheng Tao
- Mohit Bansal
- Jianfeng Gao
- Xiaogang Wang
In Representation & Self-Supervised Learning
- Jason Baldridge
- Yinfei Yang
- Daniel Matthew Cer
- Radu Soricut
- Vihan Jain
- Eugene Ie
- Soravit Changpinyo
- Nan Ding
Citations in the corpus
- On the Evaluation of Vision-and-Language Navigation Instructions (2021, positive): “Each visual feature is a concatenation of a pre-trained CNN image feature (Juan et al., 2020) with orientation vectors encoding both sine and cosine functions of the absolute and relative angles {θabs, φabs, θrel, φrel}.”
- Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “We select the top 16 box proposals and featurize each of them with Graph-RISE [34, 35], similar to [19].”
- Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, background): “, 2018), and querybased training of image models demonstrates language’s power to improve image representations (Juan et al., 2020).”