Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

By Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Hao Zhou, Minghui Qiu, Ling Shao. 2021

144 citations recorded by the source corpus.

We present a new vision-language (VL) pre-training model dubbed Kaleido-BERT , which introduces a novel kaleido strategy for fashion cross-modality representations from transformers. In contrast to random masking strategy of recent VL models, we design alignment guided masking to jointly focus more on image-text semantic relations. To this end, we carry out five novel tasks, i.e., rotation, jigsaw, camouflage, grey-to-color, and blank-to-color for self-supervised VL pre-training at patches of different scale. Kaleido-BERT is conceptually simple and easy to extend to the existing BERT framework, it attains state-of-the-art results by large margins on four downstream tasks, including text retrieval (R@1: 4.03% absolute improvement), image retrieval (R@1: 7.13% abs imv.), category recognition (ACC: 3.28% abs imv.), and fashion captioning (Bleu4: 1.2 abs imv.). We validate the efficiency of Kaleido-BERT on a wide range of e-commerical websites, demonstrating its broader potential in real-world applications.

Subfields: Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Ming-Ming Cheng
2. Deng-Ping Fan
3. Ling Shao
4. Yunchao Wei
5. Luc Van Gool
6. Ali Borji
7. Yi Yang
8. Jianbing Shen

In Large Language Models

1. Kai Yi
2. Boyang Albert Li
3. Han Guo
4. Mohamed Elhoseiny
5. P. Xu
6. Xiao Dong
7. Xunlin Zhan
8. Minlong Lu

In Multimodal Learning

1. Kai Yi
2. Boyang Albert Li
3. Guangyao Chen
4. Guangwu Qian
5. Han Guo
6. Mohamed Elhoseiny
7. P. Xu
8. Pengcheng Gao

Citations in the corpus

• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Unlike neural networks pretrained on multimodal data (e.g., [39,48,57,66,67,82,83]), our method only requires a small amount of multimodal training data and focuses on adapting linguistic knowledge learned from the textual modality.”
• Structure-Measure: A New Way to Evaluate Foreground Maps (2017, background): “& PASTE(1) [62], object detection and recognition [6, 8,31,37,63], video summarization [27], video compression [30, 32], image segmentation [69], content-based image retrieval [45, 48, 49], visual tracking, photo synthesis [13], image-text matching [87], image collection browsing [13], etc.”
• Multimodal Learning With Transformers: A Survey (2022, methodology): “In general, existing multimodal Transformer models [9], [199], [263] are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks [137], [142], [212], [249], [260], [261], [265], [266].”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “• Aligned Kaleido Patch Modeling (AKPM) is proposed for the pre-training of Kaleido-BERT [124], which contains five kaleido sub-tasks, i.e., Rotation Recognition (RR), Jig-saw Puzzle Solving (JPS), Camouflage Prediction (CP), Grey-to-Color Modeling (G2CM), and Blank-to-Color Modeling (B2CM): where…”
• GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, methodology): “In the area of fashion, [119] presented Kaleido-BERT.”
• M5Product: A Multi-modal Pretraining Benchmark for E-commercial Product Downstream Tasks (2021, methodology): “Kaleido-BERT [43] introduces a novel Kaleido strategy for fashion cross-modality representations from transformers.”