Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks

By Wen Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiangbo Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som and 1 more. Computer Vision and Pattern Recognition, 2023

533 citations recorded by the source corpus.

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEIT-3, which achieves excellent transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We use Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked “language” modeling on images (Imglish), texts (English), and image-text pairs (“parallel sentences”) in a unified manner. Experimental results show that BEIT-3 obtains remarkable performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ed8ac4ff13d32a291bbe74f3e5a138800bba47fd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Maosong Sun
2. Zhiyuan Liu
3. Yuan Yao
4. Ganqu Cui
5. Wenpeng Yin
6. Hannaneh Hajishirzi
7. Furu Wei
8. Felix Stahlberg

Citations in the corpus

• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “For a more comprehensive comparison of human datasets, we refer interested readers to [217].”
• MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, unknown): “3 of Appendix G.”
• RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, methodology): “The model uses BEiT-3 [41] as the visual module, and 13B Vicuna v1.”