UFO: A UniFied TransfOrmer for Vision-Language Representation Learning

By Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, Lijuan Wang. 2021

65 citations recorded by the source corpus.

In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e.g., image or language) or multimodal inputs (e.g., the concatenation of the image and the question), for vision-language (VL) representation learning. Existing approaches typically design an individual network for each modality and/or a specific fusion network for multimodal tasks. To simplify the network architecture, we use a single transformer network and enforce multi-task learning during VL pre-training, which includes the image-text contrastive loss, image-text matching loss, and masked language modeling loss based on the bidirectional and the seq2seq attention mask. The same transformer network is used as the image encoder, the text encoder, or the fusion network in different pre-training tasks. Empirically, we observe less conflict among different tasks and achieve new state of the arts on visual question answering, COCO image captioning (cross-entropy optimization) and nocaps (in SPICE). On other downstream tasks, e.g., image-text retrieval, we also achieve competitive performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
All in One: Exploring Unified Video-Language Pre-Training (2022, background): “More recently, UFO [36] also uses contrastive learning and employs a momentum teacher to guide the pre-training of a image-text shared encoder, which incurs large computational costs.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “During pre-training, Masked Language Modeling (MLM) and Image-Text Matching (ITM) tasks have been widely used [106, 26, 66, 127, 13, 23, 105, 54].”
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, background): “Some co-train on distinct tasks [34, 35, 36, 22] without aligning or fusing representations—effectively sharing weights across tasks—whereas others include both unimodal aspects and fused multimodal aspects for functionality in both contexts [37].”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[99] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, methodology): “We evaluate results on eight downstream datasets covering a wide range of textual and video domains ( e.g. GIFs, YouTube videos, TV shows, movies), and multiple VideoQA paradigms: open-ended VideoQA ( iVQA [113], MSRVTT-QA [109], MSVD-QA [109], ActivityNet-QA [120] and TGIF-QA FrameQA [35]), multipl”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, background): “Recent years have witnessed the rapid development of vision-language models [25, 8, 33, 24, 13, 34, 27, 30].”
MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining (2022, unknown): “Various approaches have been proposed for aligning visual and language representations, including image-text contrastive learning [23, 42], image-text matching [29, 54], masked language modeling [51, 67], and language modeling [28,61].”