nocaps: novel object captioning at scale

By Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, Peter Anderson. 2019

658 citations recorded by the source corpus.

Image captioning models have achieved impressive results on datasets containing limited visual concepts and large amounts of paired image-caption training data. However, if these models are to ever function in the wild, a much larger variety of visual concepts must be learned, ideally from less supervision. To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task. Dubbed ‘nocaps’, for novel object captioning at scale, our benchmark consists of 166,100 human-generated captions describing 15,100 images from the Open Images validation and test sets. The associated training data consists of COCO image-caption pairs, plus Open Images image-level labels and object bounding boxes. Since Open Images contains many more classes than COCO, nearly 400 object classes seen in test images have no or very few associated training captions (hence, nocaps). We extend existing novel object captioning models to establish strong baselines for this benchmark and provide analysis to guide future work.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “Instead, the dominant strategy is to start with separate language and vision models pretrained for other large-scale tasks and then learn grounding as part of task training – often resulting in myopic groundings that generalize poorly when paired visiolinguistic data is limited or biased [9, 10].”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “For captions, we estimate the cost based on nocaps [100] (1), that follows a similar data collection protocol as COCO.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “The results in Table 2 (f) show that the variants of Oscar consistently outperform the previous SoTA method UpDown [1].”
Towards Language-Free Training for Text-to-Image Generation (2021, background): “These works aim at learning generic multi-modal representations for downstream tasks like visual question answering [2, 14], image captioning [1, 27], visual commonsense reasoning [55].”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “Additionally, we use the NoCaps (Agrawal et al., 2019) dataset to test the model performance when there are outof-domain objects.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Image Captioning We consider two datasets for image captioning: NoCaps (Agrawal et al., 2019) and COCO, both evaluated using the model finetuned on COCO with the LM loss.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “We evaluate on two datasets – CoCo Captions [9] and NoCaps [1], measuring performance using the BLEU@4 and CIDEr metrics.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Recent advances in image captioning [1, 5, 35] can be largely attributed to vision-language pre-training (VLP) [26, 30, 37, 40], the current prevailing training paradigm for vision-language (VL) research.”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The dataset is COCO [31] with Karpathy split [24], and the model is also evaluated against the val and test sets of the nocaps [1] benchmark.”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “We use the COCO-captions [7,22], nocaps [1] , and Conceptual Captions [33] datasets.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “, 2015), NoCaps (Agrawal et al., 2019), and Multi30k (Elliott et al.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “This problem is distinct from the novel object captioning problem [1,23], which relies on abundant in-domain data but zero out-of-domain data.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “…uses the priming of annotators to mention one or more of displayed ﬁne-grained ground-truth object classes (e.g., “red panda”) present in the image [2], the large-scale and natural ﬁne-grainedness of CC12M suc-ceeds in correctly learning to generate captions containing such concepts, in spite of…”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, background): “Numerous tasks have been proposed to combine vision and language to enhance the understanding of either or both, such as video/image captioning [18, 60, 2], visual question answering (VQA) [4], and natural language moment retrieval [25], etc.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, positive): “We perform finetuning on COCO, and evaluate on both COCO test set and zero-shot transfer to NoCaps (Agrawal et al., 2019) of-the-art performance with significant improvement on NoCaps over existing methods, demonstrating strong generalization ability to out-domain images.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “…on the widely-used Karpathy split (Karpathy & Li, 2015) of COCO (Lin et al., 2014) and Flickr30K (Young et al., 2014), the COCO test set, nocaps (Agrawal et al., 2019)2 which focuses on novel objects, TextCaps (Sidorov et al., 2020) which focuses on scene-text understanding, and VizWiz-Captions…”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “They aim to improve downstream tasks such as visual question answering [2, 27], image captioning [1, 41], visual commonsense reasoning [70].”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “For image captioning, we apply simple cross-entropy loss (same as the captioning loss used in pretraining) and finetune the model on the training split of MSCOCO to predict for MSCOCO test split and NoCaps online evaluation.”
VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “(iv) The Novel Object Captioning (NoCaps) results are shown in Table 9.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “Similarly, Constrained Beam Search (Anderson et al. 2017) is exploited to generate captions that contain detected novel objects (Agrawal et al. 2019).”