GIT: A Generative Image-to-text Transformer for Vision and Language
By Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Qinghong Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 2022
787 citations recorded by the source corpus.
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at \url{https://github.com/microsoft/GenerativeImage2Text}.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Multimodal Learning
In Natural Language Generation
Citations in the corpus
- Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Apart from textual instructions, LLaMA-Adapter is capable of answering a question based on input of other modalities, which augments the language model with rich cross-modal information.”
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, background): “Recent years have seen significant advancement for computer vision, thanks to improved network architecture [4, 9, 12,28], large-scale model training [8,29,35], and other factors.”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “KOSMOS-1 significantly outperforms GIT [WYH22] by 4.”
- Generalized Decoding for Pixel, Image, and Language (2022, background): “This de-sign makes it intractable not only for global image-text contrastive learning [59, 79], but also generative pretraining [70].”
- SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “While we utilized EnCodec due to its accessibility and prior usage, future work should address this issue by developing an acoustic tokenization model more suitable for handling speech under various acoustic conditions.”
- PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “GIT2 (Wang et al., 2022a) has demonstrated strong performance without the OCR input, while PaLI-17B shows the superiority of levaraging a specialized OCR system for a better recipe to solve these tasks.”
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, methodology): “…models are those specifically designed for localization tasks ( e.g. , UNINEXT, Yan et al., 2023 and G-DINO, Liu et al., 2023b), or generalist pretraining models that have undergone multitask localization finetuning ( e.g. , Yang et al., 2022) or single-task finetuning ( e.g. , Wang et al., 2022b).”
- OpenAGI: When LLM Meets Domain Experts (2023, background): “• Vision-Language Models: Visual Question Answering (VQA) involves answering questions based on an image [37]; Image Captioning generates textual descriptions of the visual content depicted in an image; Text-to-Image Generation generates images from a given input sentence or sequence of words [28].”
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, neutral): “Vision-language models The general pre-trained vision-language model are GIT-large [57], VILT-B/32 [28], OFA-large [58], and mPLUG-large [31].”
- Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, background): “For natural images, recent work— GIT2 (Wang et al., 2022a) and PaLI (Chen et al., 2022c)— Figure 2: Comparison of our variable resolution inputs and the typical fixed resolution input.”
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, methodology): “The development of Vision-Language Models (VLMs) has transitioned from being rooted in BERT-based language decoders [13, 31, 47] for merging visual and textual data [34, 36, 59, 64], to a notable advancement ushered by the integration of Large Language Models (LLMs) [4, 7, 10, 11, 17, 53, 61–63].”
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, neutral): “2”
- GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, background): “Many works have focused on improving web agents relying on the HTML document (Deng et al., 2023; Gur et al., 2023; 2022; 2023; Kim et al., 2023; Sridhar et al., 2023).”
- Sigmoid Loss for Language Image Pre-Training (2023, methodology): “Seminal works CLIP [34] and ALIGN [23] established the viability of this approach at a large scale, and following their success, many large image-text datasets became available privately [55, 13, 21, 45] and publicly [37, 6, 15, 7, 38].”