Pre-training image-language transformers for open-vocabulary tasks
By A. Piergiovanni, Weicheng Kuo, Anelia Angelova. 2022
13 citations recorded by the source corpus.
We present a pre-training approach for vision and language transformer models, which is based on a mixture of diverse tasks. We explore both the use of image-text captioning data in pre-training, which does not need additional supervision, as well as object-aware strategies to pre-train the model. We evaluate the method on a number of text-generative vision+language tasks, such as Visual Question Answering, visual entailment and captioning, and demonstrate large gains over standard pre-training methods.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Cordelia Schmid
- Andrew Zisserman
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
In Large Language Models
- Yi Tay
- Mostafa Dehghani
- Neil Houlsby
- Arsha Nagrani
- Anurag Arnab
- Mandar Joshi
- Radu Soricut
- Lucas Beyer
In Multimodal Learning
- Neil Houlsby
- Radu Soricut
- Soravit Changpinyo
- Matthias Minderer
- Bo Pang
- Arsha Nagrani
- Xi Chen
- Basil Mustafa
Citations in the corpus
- PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “• English-only Object-Aware (OA) VQA is based on VQA triplets derived from automatically-produced, non-exhaustive object labels, inspired by Piergiovanni et al. (2022a).”
- PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “The pretraining mixture consists of the following data and objectives: (i) span corruption on textonly data (15% of tokens); (ii) split-captioning on WebLI alt-text data [21, 5]; (iii) captioning on CC3M [22] on native and translated alt-text data (over the same 35 languages covered by XM3600 [23]);”