ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

By Yuhao Cui, Zhou Yu, Chunqi Wang, Zhongzhou Zhao, Ji Zhang, Meng Wang, Jun Yu. 2021

60 citations recorded by the source corpus.

Vision-and-language pretraining (VLP) aims to learn generic multimodal representations from massive image-text pairs. While various successful attempts have been proposed, learning fine-grained semantic alignments between image-text pairs plays a key role in their approaches. Nevertheless, most existing VLP approaches have not fully utilized the intrinsic knowledge within the image-text pairs, which limits the effectiveness of the learned alignments and further restricts the performance of their models. To this end, we introduce a new VLP method called ROSITA, which integrates the cross- and intra-modal knowledge in a unified scene graph to enhance the semantic alignments. Specifically, we introduce a novel structural knowledge masking (SKM) strategy to use the scene graph structure as a priori to perform masked language (region) modeling, which enhances the semantic alignments by eliminating the interference information within and across modalities. Extensive ablation studies and comprehensive analysis verifies the effectiveness of ROSITA in semantic alignments. Pretrained with both in-domain and out-of-domain datasets, ROSITA significantly outperforms existing state-of-the-art VLP methods on three typical vision-and-language tasks over six benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Wei Wang
6. Fei Huang
7. Luo Si
8. Haiyang Xu

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Fei Huang
6. Wei Wang
7. Haiyang Xu
8. Luo Si

In Multimodal Learning

1. Fei Huang
2. Chenliang Li
3. Haiyang Xu
4. Ming Yan
5. Ji Zhang
6. Weihua Chen
7. Fan Wang
8. Songfang Huang

Citations in the corpus

• Achieving Human Parity on Visual Question Answering (2021, background): “• ROSITA [13]: different from other VLP methods that only exploit the intra-modal knowledge from eachmodality, it encodes the object-level cross-modal knowledge and intra-modal knowledge in a unified scene graph, so as to better learn the fine-grained semantic alignments across modalities.”
• Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “ment [43], [44], [45], relation alignment [46], [47] tasks to optimize the pre-training process.”