VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training

By Xiaowei Hu, Xi Yin, Kevin Qinghong Lin, Lijuan Wang, L. Zhang, Jianfeng Gao, Zicheng Liu. 2020

58 citations recorded by the source corpus.

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other than COCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pre-training (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, background): “Recent studies [61, 34, 21] unified the pre-training framework to cover generation tasks, e.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “SOTA model SimVLM [55] PICa [58] CFR [33] Pythia [46] SimVLM SimVLM VIVO [22] SimVLM OSCAR [29]”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Since the birth of ViLBERT [30] and LXMERT [40], we have witnessed a boom of methods for vision-language pre-training [6,7,13,22,26, 37, 45, 46].”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “…model ( ic pre-trained on CC3M+CC12M) to existing state-of-the-art results on no-caps, and show that ours achieves state-of-the-art performance on CIDEr, outperforming a concurrent work [31] that uses a different pre-training approach directly on the Open Images dataset, which nocaps is based on.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “The reason could be that LEMON uses MLM, instead of LM, which might be more difficult to train.”
VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “SoTAs: VQA is from ERNIE-VIL [43], GQA is from NSM [12], NoCaps is from VIVO [9], NLVR2 is from VILLA [7], the rest tasks are from OSCAR [21].”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, methodology): “Remarkable progress [32, 43, 12, 25, 22, 23, 10, 3] has been made recently on visionlanguage tasks through network pre-training on massive data with image-text pairs.”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “Following the prominent progress in the transformer-based [68] pretraining in natural language [14, 51, 38, 5, 11, 52], visual-linguistic pre-training models, either for image+text [44, 64, 9, 42, 26, 81, 41, 18, 40, 45] or for video+text [61, 40, 46, 84, 39].”
VinVL: Revisiting Visual Representations in Vision-Language Models (2021, methodology): “SoTAs: VQA is from ERNIE-VIL [42], GQA is from NSM [11], NoCaps is from VIVO [9], NLVR2 is from VILLA [7], the rest tasks are from O SCAR [20].”
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, methodology): “Pre-trained models [39, 75, 59, 35, 37, 24] such as ViLBERT [43], LXMERT [64] and UNITER [13] have demonstrated great generalizability over diverse V+L tasks [80, 23, 47, 70, 4, 22, 46, 9], such as Visual Question Answering (VQA) [6], Visual Commonsense Reasoning (VCR) [78], and Referring Expression”
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021, unknown): “VIVO (Hu et al., 2020) targets novel object captioning.”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “Recent studies have witnessed its great development which are primarily reflected in the aspects of more advanced cross-modal fusion architectures [11, 45, 50, 54, 57, 63, 65, 67, 70]; more expressive object-centric features [4, 69] & tags [16, 23, 35, 58] obtained from a pre-trained object detectio”
Playing Lottery Tickets with Vision and Language (2021, unknown): “This pretrain-thenfinetune learning paradigm has also been extended to diverse V+L tasks such as novel object captioning [26], visual dialog [51, 71], vision-language navigation [24, 49, 25], OCR-based VQA [73], robust VQA [38], and video-text pre-training [64, 63, 37, 82, 35].”
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, methodology): “In the same spirit as in image captioning literature [21, 28], we use a causal self-attention mask where a caption token can only attend to the existing output tokens.”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Since then, we have witnessed a boom of VLP methods [36, 33, 63, 79, 24, 77, 9, 38].”