SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

By Chenliang Li, Ming Yan, Haiyang Xu, Fuli Luo, Wei Wang, Bin Bi, Songfang Huang. 2021

41 citations recorded by the source corpus.

Vision-language pre-training (VLP) on large-scale image-text pairs has recently witnessed rapid progress for learning cross-modal representations. Existing pre-training methods either directly concatenate image representation and text representation at a feature level as input to a single-stream Transformer, or use a two-stream cross-modal Transformer to align the image-text representation at a high-level semantic space. In real-world image-text data, we observe that it is easy for some of the image-text pairs to align simple semantics on both modalities, while others may be related after higher-level abstraction. Therefore, in this paper, we propose a new pre-training method SemVLP, which jointly aligns both the low-level and high-level semantics between image and text representations. The model is pre-trained iteratively with two prevalent fashions: single-stream pre-training to align at a fine-grained feature level and two-stream pre-training to align high-level semantics, by employing a shared Transformer network with a pluggable cross-modal attention module. An extensive set of experiments have been conducted on four well-established vision-language understanding tasks to demonstrate the effectiveness of the proposed SemVLP in aligning cross-modal representations towards different semantic granularities.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ishan Misra
2. Mohit Bansal
3. Jianfeng Gao
4. Gabriel Synnaeve
5. Yoshua Bengio
6. Yann LeCun
7. Yejin Choi
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Yi Yang
2. Luc Van Gool
3. Ishan Misra
4. Trevor Darrell
5. Xiaogang Wang
6. Dacheng Tao
7. Jitendra Malik
8. Ming-Hsuan Yang

In Multimodal Learning

1. Fei Huang
2. Chenliang Li
3. Mannat Singh
4. Haiyang Xu
5. Yann LeCun
6. Guangyao Chen
7. Guangwu Qian
8. Ishan Misra

In Representation & Self-Supervised Learning

1. Ishan Misra
2. Yann LeCun
3. Xiao Wang
4. Gabriel Synnaeve
5. Mannat Singh
6. Fei Huang
7. Chenliang Li
8. Haiyang Xu

Citations in the corpus

• MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, background): “While this approach has improved state of the art results on a wide variety of tasks such as image-text retrieval [65], phrase grounding [22], image captioning [24] and visual question answering [21], it leaves opportunity for a more tightly knit architecture, such as MDETR, in which information flo”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “77 SemVLP [187] arXiv-2021 image-text Trans MLM, MOP, ITM, QA Fuse the single- and two-stream architectures 2.”
• Multimodal Learning With Transformers: A Survey (2022, unknown): “[196] C. Li et al., “SemVLP: Vision-language pre-training by aligning semantics at multiple levels,” 2021, arXiv:2103.07829.”
• Data Efficient Masked Language Modeling for Vision and Language (2021, background): “Some models also add the visual question answering objective during the pretraining phase (Tan and Bansal, 2019; Li et al., 2021).”
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021, methodology): “Besides, SemVLP (Li et al., 2021) is pre-trained iteratively with two prevalent fashions.”