InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining

By Junyang Lin, Yichang Zhang, Jie Liu, Jingren Zhou, Hongxia Yang. 2020

91 citations recorded by the source corpus.

Multi-modal pretraining for learning high-level multi-modal representation is a further step towards deep learning and artificial intelligence. In this work, we propose a novel model, namely InterBERT (BERT for Interaction), which owns strong capability of modeling interaction between the information flows of different modalities. The single-stream interaction module is capable of effectively processing information of multiple modalilties, and the two-stream module on top preserves the independence of each modality to avoid performance downgrade in single-modal tasks. We pretrain the model with three pretraining tasks, including masked segment modeling (MSM), masked region modeling (MRM) and image-text matching (ITM); and finetune the model on a series of vision-and-language downstream tasks. Experimental results demonstrate that InterBERT outperforms a series of strong baselines, including the most recent multi-modal pretraining methods, and the analysis shows that MSM and MRM are effective for pretraining and our method can achieve performances comparable to BERT in single-modal tasks. Besides, we propose a large-scale dataset for multi-modal pretraining in Chinese, and we develop the Chinese InterBERT which is the first Chinese multi-modal pretrained model. We pretrain the Chinese InterBERT on our proposed dataset of 3.1M image-text pairs from the mobile Taobao, the largest Chinese e-commerce platform. We finetune the model for text-based image retrieval, and recently we deployed the model online for topic-based recommendation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Lijuan Wang
4. Linjie Li
5. Zicheng Liu
6. Jianfeng Wang
7. Xiaowei Hu
8. Jianwei Yang

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Zhe Gan
3. Luc Van Gool
4. Lijuan Wang
5. Zicheng Liu
6. Dacheng Tao
7. Cordelia Schmid
8. Ling Shao

In Large Language Models

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Jianfeng Gao
5. Haifeng Wang
6. Ming Ding
7. Pengchuan Zhang
8. Yejin Choi

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Jianfeng Gao
6. Xiaowei Hu
7. Linjie Li
8. Jianwei Yang

Citations in the corpus

• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, methodology): “So their performance will drop dramatically when applied to language tasks (Lin et al., 2020b), which has also been revealed by our experiments (see The unified-modal architecture mainly has the following advantages compared with previous methods: • We can utilize large scale of non-paired text…”
• M6: A Chinese Multimodal Pretrainer (2021, methodology): “A number of studies [4, 11, 17, 22, 24, 25, 28, 29, 38, 51] have created new state-of-the-art performances for various cross-modal downstream tasks.”
• VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “(i) The VQA results are shown in Table 5, where our single OSCAR+B model outperforms the best ensemble model (InterBERT large [22]) on the VQA leaderboard as of Dec.”
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “Multimodal Pretraining Multimodal pretraining has been developing rapidly [38, 13, 39, 40, 14, 41, 42, 43, 44, 15, 16, 17, 45, 46, 47].”
• MiniVLM: A Smaller and Faster Vision-Language Model (2020, background): “Remarkable progress [32, 43, 12, 25, 22, 23, 10, 3] has been made recently on visionlanguage tasks through network pre-training on massive data with image-text pairs.”
• VinVL: Revisiting Visual Representations in Vision-Language Models (2021, methodology): “(i) The VQA results are shown in Table 5, where our single OSCAR+B model outperforms the best ensemble model (InterBERT large [21]) on the VQA leaderboard as of Dec.”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “Different from regular negative image-text samples, the authors of InterBERT [115] design the image-text matching with hard negatives (i.”
• VLP: A Survey on Vision-language Pre-training (2022, background): “PREVALENT[117] Image CNNGFs+Xformer Xformer Single-stream No MLM+MVM Matterport3D VLN”
• Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays (2021, background): “A variety of learning objectives are designed in addition to the masked word prediction task, e.g ., masked region feature regression [7, 40], text generation [33, 11, 5, 8], and object classification [34, 24].”
• Deep Multimodal Data Fusion (2024, methodology): “Transformer-based large pre-trained models have been dominant on lots of multimodal data fusion tasks, such as interBERT [101] and videoBERT [168].”
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain (2021, background): “In contrast to the boom in generic tasks ( e.g ., VCR [9, 43, 78], VQA [32, 38]), other tasks such as visual relationship detection (RVL-BERT [10]), visual navigation ( i.e ., PERVALENT [24] and VLN-BERT [48]), and visual dialog ( e.g ., VisualD [50], VD-BERT [70]) are still in their infancy.”
• Multimodal Learning With Transformers: A Survey (2022, methodology): “Thus, multimodal Transformers can be divided into single-stream (e.g., Uniter [106], Visualbert [104], Vl-bert [105], Unified VLP [110]), multi-stream (e.g., ViLBERT [102], Lxmert [103], ActBERT [114]), hybrid-stream (e.g., InterBERT [188]), etc.”
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “Large-scale multimodal pre-training has taken innovative strides in the realm of vision-and-language (V+L) research [9, 40, 44, 58, 62, 61, 36, 82, 38].”