An Empirical Study of Training End-to-End Vision-and-Language Transformers

By Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Nanyun Peng, Zicheng Liu, Michael Zeng. 2021

462 citations recorded by the source corpus.

Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be more efficient than previous region-feature-based methods, their performance on downstream tasks often degrades significantly. In this paper, we present Meter, a Multimodal End-to-end TransformER framework, through which we investigate how to design and pre-train a fully transformer-based VL model in an end-to-end manner. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, De-BERTa), multimodal fusion module (e.g., merged attention vs. co-attention), architectural design (e.g., encoder-only vs. encoder-decoder), and pre-training objectives (e.g., masked image modeling). We conduct comprehensive experiments and provide insights on how to train a performant VL transformer. Meterachieves an accuracy of 77.64% on the VQAv2 test-std set using only 4M images for pre-training, surpassing the state-of-the-art region-feature-based model by 1.04%, and outperforming the previous best fully transformer-based model by 1.6%. Notably, when further scaled up, our best VQA model achieves an accuracy of 80.54%. Code and pre-trained models are released at https://github.com/zdou0830/METER.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Model Architectures

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, methodology): “We extended the learned feature representation along space (from scene to objects) using the dynamic head adapter (Dai et al., 2021a), time (from static image to videos) via proposed video CoSwin adapter, and modality (from images to language) via METER adapter (Dou et al., 2021).”
Generalized Decoding for Pixel, Image, and Language (2022, background): “The field has evolved from a transformer fusion model [10, 46, 96] with pre-extracted object features [1] to end-to-end transformers [21, 36, 43], that directly learn from raw image pixels.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “We use the image encoder pre-trained with contrastive tasks because recent studies show superior performance with such image encoder, e.g. Yuan et al. (2021); Dou et al. (2021); Alayrac et al. (2022).”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, background): “Therefore, CoCa can perform both image captioning and multimodal understanding downstream tasks without any further fusion adaptation (Shen et al., 2021; Dou et al., 2021).”
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022, methodology): “Following [11, 26], we use a visual transformer [27] directly on the image patches as the visual encoder, which is more computation-friendly than using pre-trained object detectors for visual feature extraction [8, 9].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “For example, the ALBEF [130] and METER [157] achieves 82.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Similarly, a METER [56] head is used for vision language representation, and CSwin [57] is used for video-based understanding.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “Despite the impressive performance of pretrained vision and language transformers on a wide variety of multimodal tasks [47,51,56], they remain poorly understood [8,19,46, 67].”
A Cookbook of Self-Supervised Learning (2023, background): “While its advantages in vision-language downstream applications are evident [Shen et al., 2022, Dou et al., 2022], shared embedding spaces can also be constructed by training solely the vision encoder first, fixing it, and then training a matching language encoder, as described in [Zhai et al.,…”
Open-vocabulary Attribute Detection (2022, background): “[13] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al.”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Following previous work [6, 28, 32, 13, 64, 66], we perform coarse-grained pre-training on COCO [41], Conceptual Captions [56], SBU Captions [47], and Visual Genome [29].”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, unknown): “Following the success of image-text pretraining [14,21,23,24,28,34,36,39–41,53– 55,58,60,68,69,85,88,92,93,99,118–120,124,129], recent works have explored video-text pretraining [3–5,26,31,32, 43,49,52,56,71,72,74,80,81,89,96,98,108,111,111–115, 121, 122].”
VLP: A Survey on Vision-language Pre-training (2022, methodology): “To mitigate the distribution biases between pretraining and funetuning, DeVLBert [32] is proposed to perform intervention-based learning.”
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts (2021, methodology): “Compared to METER-Swin (Dou et al., 2021) which also uses Swin Transformer as the image encoder, X-VLM has better performance.”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “Thus, we use cross-attention [22,66] to the image features v w,h during this task.”