GIT: A Generative Image-to-text Transformer for Vision and Language

By Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Qinghong Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang. 2022

787 citations recorded by the source corpus.

In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at \url{https://github.com/microsoft/GenerativeImage2Text}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Sergey Levine
  4. Furu Wei
  5. Quoc V. Le
  6. Yoshua Bengio
  7. Percy Liang
  8. Zhe Gan

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Jianfeng Gao
  4. Zicheng Liu
  5. Linjie Li
  6. Jianfeng Wang
  7. Chunyuan Li
  8. Jianwei Yang

In Natural Language Generation

  1. Huan Sun
  2. Li Dong
  3. Radu Soricut
  4. Ming-Wei Chang
  5. Jianfeng Gao
  6. Arsha Nagrani
  7. Shujie Liu
  8. Zhengyuan Yang

Citations in the corpus