Microsoft COCO Captions: Data Collection and Evaluation Server

By Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll'ar, C. Lawrence Zitnick. 2015

2,946 citations recorded by the source corpus.

In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Distributed Representations of Sentences from Unlabelled Data (2016, methodology): “CaptionRep Using the same overall architecture, we trained ( BOW and RNN) models to map captions in the COCO dataset (Chen et al., 2015) to pre-trained vector representations of images.”
The E2E Dataset: New Challenges For End-to-End Generation (2017, methodology): “Detailed system training parameters are given in the supplementary material. 5To measure the scores, we used slightly adapted versions of the ofﬁcial MT-Eval script (BLEU, NIST) and the COCO Caption (Chen et al., 2015) metrics (METEOR, ROUGEL, CIDEr). All evaluation scripts used here are available a”
VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “Therefore we reach to a resource of paired data: COCO (Chen et al., 2015) that contains images each paired with 5 independent captions.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “We train and evaluate on the VQA 2.0 dataset [3] consisting of 1.1 million questions about COCO images [5] each with 10 answers.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “Training Details: We train on the train2017 split of the COCO Captions dataset [35], which provides 118K images with five captions each.”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “UNITER[21] combines four datasets (Conceptual Captions[2], SBU Captions[3], Visual Genome[22] and MSCOCO[5]) together to form a 9.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “When ﬁne-tuning on image and sentence retrieval tasks, our experiment results on Flickr30K and MSCOCO datasets demonstrate that our pre-trained Transformer model can boost retrieval performance signiﬁcantly.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Wepresent experiments on standard benchmark datasets for sentencebased image description:MSCOCODataset [3] and Flickr30K dataset [42] to evaluate the performance of our proposed framework.”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, background): “Keywords Computer Vision · Dataset · Image · Scene Graph · Question Answering · Objects · Attributes · Relationships · Knowledge · Language · Crowdsourcing”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “Prominent tasks include visual question answering (Antol et al., 2015), image cap-tioning (Chen et al., 2015), vision-language navigation (Anderson et al., 2018a), image-text retrieval (Wang et al., 2016) and so on.”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “To calculate the standard metrics we use the official evaluation code provided with the COCO captions dataset (Chen et al., 2015).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “Across a number of domains spanning vision, language, and audio modalities – and via a small amount of creative prompt-enabled multimodal composition – SMs are quantitatively competitive with zero-shot state-of-the-art on standard benchmarks including (i) image captioning on MS COCO [15, 16], (ii) c”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “We conduct our experiments and analysis on four diverse image-text tasks: VQAv2 [10], GQA [19], NLVR(2) [46], and MSCOCO captioning [5].”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “We evaluate on two datasets – CoCo Captions [9] and NoCaps [1], measuring performance using the BLEU@4 and CIDEr metrics.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, positive): “COCO Caption and Flickr30k are the typical datasets used to evaluate the performance on image caption and image-text retrieval. classiﬁcation, video classiﬁcation, image-text retrieval, and image caption.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “The dataset consists of 15100 images from Open Images, and covers more than 600 object categories, of which nearly 400 of them are unseen from the training set in COCO [5].”