Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling

By Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, Lijuan Wang. 2021

39 citations recorded by the source corpus.

Subfields: Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Zhe Gan
2. Lijuan Wang
3. Linjie Li
4. Jianfeng Gao
5. Zicheng Liu
6. Cordelia Schmid
7. Jianfeng Wang
8. Xiaowei Hu

Citations in the corpus

• GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “For the scene-text related VQA tasks, existing approaches (Yang et al., 2021c; Hu et al., 2020) typically leverages the OCR engine to generate the scene text and use dynamic pointer network to decide the current output token should be OCR or the general text.”
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, methodology): “Following this idea, [56] and [57] demonstrate text-generation-based multimodal pretrained models.”
• Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) (2022, background): “Following this trend, recent advances started to focus on unifying large-scale language tasks [1, 51, 67] or cross-modality applications [6, 66, 71] through a shared sequence-to-sequence framework, where different types of tasks and modalities are all expressed in the format of natural language.”
• GLIPv2: Unifying Localization and Vision-Language Understanding (2022, background): “Recently, a general interest arises in building general-purpose vision systems [21, 24, 56, 42], also called vision foundation models [6, 57], that solve various vision tasks simultaneously, such as image classification [30], object detection [39], and Visual-Language (VL) understanding [3, 11, 27].”
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Recent works [11– 13, 15, 17, 44, 59, 101, 117, 132] have shown that it is possible to cast various computer vision problems as a language modeling task, addressing object detection [11], grounded image captioning [117] or visual grounding [132].”
• VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training (2022, methodology): “Notably, our model can achieve a CIDEr score of 127.7 on the MSCOCO test set using only 110k images for”