Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

By Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som and 1 more. 2022

724 citations recorded by the source corpus.

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked"language"modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “With the vast advancement of the Transformer architecture [35, 189] in both CV and NLP, there is a trend to pre-train large-scale unified vision-language (VL) models [1, 27, 105, 196, 197, 200] to perform vision grounded text generation tasks, such as image captioning and visual question answering (”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Recent multi-modal vision-language models have up to 1.9B parameters (e.g., BeiT-3 [62]).”
Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
Generalized Decoding for Pixel, Image, and Language (2022, background): “[78] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, neutral): “1 — Mask2Former‡‡ [12] BEiT-3†† [52] — — — 896×896 — — — 62.”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “Inspired by the success of BERT [10], ViL-BERT [43], LXMERT [59], UNITER [6], Oscar [36], ALBEF [35], and BEiT [64] further push the boundary of multimodal reasoning.”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “GIT2 (Wang et al., 2022a) has demonstrated strong performance without the OCR input, while PaLI-17B shows the superiority of levaraging a specialized OCR system for a better recipe to solve these tasks.”
PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents (2023, background): “…of foundational models has been the main driving force in artiﬁcial intelligence, for example, large language models [26,8,2,23] trained with either autoregressive prediction or masked token inpainting, and computer vision models [17,25,30] trained by contrasting visual-language features.”
EVA-02: A Visual Representation for Neon Genesis (2023, methodology): “Recent research advancements have led to a surge of interest in scaling up vision [81, 44, 124, 17] as well as vision-language [140, 123, 30, 139] representations.”
PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, neutral): “Skin Tone 1 [2] 2 [871] 3 [3008] 4 [522] 5 [184] 6 [85] 7 [54] 8 [49] 9 [6] 10 [1] 0.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “Many models have been proposed for tasks with text and/or image input and text output (Cho et al., 2021; Wang et al., 2022d; Li et al., 2022b; Wang et al., 2021; Kaiser et al., 2017; Sun et al., 2022; Chen et al., 2022d; Wang et al., 2022c).”
ClimaX: A foundation model for weather and climate (2023, neutral): “Current notable examples include BERT (Devlin et al., 2018), GPT (Brown et al., 2020) and PaLM (Chowdhery et al., 2022), in language, CLIP (Radford et al., 2021), Florence (Yuan et al., 2021), BEiT (Wang et al., 2022) for vision-language.”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023, background): “Among these efforts, Alayrac et al. (2022b); Wang et al. (2022); Huang et al. (2023b); Xu et al. (2023b); Zhang et al. (2023b); Sun et al. (2023) pre-train multi-modal LLMs with massive interleaved image-text data or speech-text data to accommodate multi-modal input.”
Multimodal Learning With Transformers: A Survey (2022, background): “In particular, the recent success of large language models and their multimodal derivatives [82], [83], [84], [85], [86] further demonstrates the”