UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling

By Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, Lijuan Wang. 2021

147 citations recorded by the source corpus.

. We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indicate word-box alignments in the sequence. UniTAB thus could provide a more comprehensive and interpretable image description, by freely grounding generated words to object regions. On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations. On general VL tasks that have different desired output formats ( i.e ., text, box, or their combination), UniTAB with a single network achieves better or comparable performance than task-specific state of the art. Experiments cover 7 VL benchmarks, including grounded captioning, visual grounding, image captioning, and visual question answering. Furthermore, UniTAB’s unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, background): “Furthermore, X-Decoder also beats other general-purpose models like UniT [32], GPV [27], UniTAB [85] and Unified-IO [54].”
Ferret: Refer and Ground Anything Anywhere at Any Granularity (2023, result): “Our work is also related to previous work that aims to unify text and bounding box output for vision-language (VL) models, such as UniTAB (Yang et al., 2022), OFA (Wang et al., 2022b), and Unified-IO (Lu et al., 2022), which also represent bounding boxes using a set of additional discrete tokens as ”
i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “Baseline methods include image captioning models BUTD (Anderson et al., 2018) and AoANet (Huang et al., 2019); vision-language generative models, e.g., VL-BART, VL-T5 (Cho et al., 2021), XGPT(Xia et al., 2021); models using additional auxiliary input such as UNITAB (Yang et al., 2022) with object de”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, methodology): “…models are those speciﬁcally designed for localization tasks ( e.g. , UNINEXT, Yan et al., 2023 and G-DINO, Liu et al., 2023b), or generalist pretraining models that have undergone multitask localization ﬁnetuning ( e.g. , Yang et al., 2022) or single-task ﬁnetuning ( e.g. , Wang et al., 2022b).”
ReCo: Region-Controlled Text-to-Image Generation (2022, background): “By quantizing coordinates [3, 40, 42], we can represent the region by four discrete position tokens P , <x1>,<y1>,<x2>,<y2>, arranged as a sequence similar to a short natural language sentence.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, methodology): “The fusion encoder studies [10, 12, 25, 32, 38, 60, 67, 71, 77] take image and text features as inputs, and design early fusion methods for VL interaction.”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “Unified-IO [33], Pix2Seq v2 [8], and UniTab [63] extend this idea by using discrete coordinate tokens to encode and decode spatial information for more tasks.”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “Vision-language models [13, 31, 51, 14, 44, 36, 72, 69, 74, 25] approach multimodal intelligence of jointly understanding and generating vision and language signals.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, background): “Alternatively, image locations can be encoded as special tokens in the input/output text (Yang et al., 2021; Yao et al., 2022; Zhu et al., 2022a) following Chen et al. (2022b).”