Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining

By Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, Xiaodan Liang. 2021

87 citations recorded by the source corpus.

Nowadays, customer’s demands for E-commerce are more diversified, which introduces more complications to the product retrieval industry. Previous methods are either subject to single-modal input or perform supervised image-level product retrieval, thus fail to accommodate real-life scenarios where enormous weakly annotated multi-modal data are present. In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories. To promote the study of this challenging task, we contribute Product1M, one of the largest multi-modal cosmetic datasets for real-world instance-level retrieval. Notably, Product1M contains over 1 million image-caption pairs and consists of two sample types, i.e., single-product and multi-product samples, which encompass a wide variety of cosmetics brands. In addition to the great diversity, Product1M enjoys several appealing characteristics including fine-grained categories, complex combinations, and fuzzy correspondence that well mimic the real-world scenes. Moreover, we propose a novel model named Cross-modal contrAstive Product Transformer for instance-level prodUct REtrieval (CAPTURE), that excels in capturing the potential synergy between multi-modal inputs via a hybrid-stream transformer in a self-supervised manner. CAPTURE generates discriminative instance features via masked multi-modal learning as well as cross-modal contrastive pretraining and it outperforms several SOTA cross-modal baselines. Extensive ablation studies well demonstrate the effectiveness and the generalization capacity of our model. Dataset and codes are available at https: //github.com/zhanxlin/Product1M.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiaodan Liang
2. Yunchao Wei
3. Hang Xu
4. Zhenguo Li
5. Yao Zhao
6. Lu Hou
7. Xin Jiang
8. Lewei Yao

In Computer Vision and Pattern Recognition

1. Yunchao Wei
2. Xiaodan Liang
3. Zhenguo Li
4. Hang Xu
5. Yi Yang
6. Yao Zhao
7. Yonghong Tian
8. Chunjing Xu

In Information Systems

1. Xin Jiang
2. Yao Zhao
3. Yonghong Tian
4. Xiaodan Liang
5. Yaowei Wang
6. Zhenguo Li

In Information Retrieval & Search

1. Yao Zhao
2. Yunchao Wei
3. Zhenguo Li

In Multimodal Learning

1. Guangyao Chen
2. Guangwu Qian
3. P. Xu
4. Pengcheng Gao
5. Xiao Dong
6. Xiao Wang
7. Xunlin Zhan
8. Xiaoyong Wei

Citations in the corpus

• FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, background): “(1) One line of work (Chen et al., 2020; Li et al., 2020b; Dong et al., 2021; Li et al., 2021b; Zhang et al., 2021; Zhan et al., 2021) uses a pre-trained object detector to extract region-of-interest (ROI) features from images, and then fuses it with the paired text through a VLP model.”
• Multimodal Learning With Transformers: A Survey (2022, methodology): “, masked language modelling (MLM) [7], [137]) and supervised (e.”
• M5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream Tasks (2021, background): “Meanwhile, in several product datasets (e.g. RPC checkout [5] (200 categories), Dress Retrieval [6] (no categories) and Product1M [7] (458 categories)), the category number is limited, which cannot cover more categories for the performance verification of downstream tasks.”
• CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, methodology): “For multi-modal retrieval, we adopt mean Average Precision (mAP@N ) [68, 75, 15] as the evaluation metrics.”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “It can be used for a variety of tasks, such as the image-text, text-text, and image-image retrieval, etc. • Product1M [98] contains 1,182,083 image-caption pairs, 458 categories, 92,200 instance.”