Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

By Zarana Parekh, Jason Baldridge, Daniel Matthew Cer, Austin Waters, Yinfei Yang. 2020

74 citations recorded by the source corpus.

By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning. Unfortunately, datasets have limited cross-modal associations: images are not paired with other images, captions are only paired with other captions of the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines research into how inter-modality learning impacts intra-modality tasks. We address this gap with Crisscrossed Captions (CxC), an extension of the MS-COCO dataset with human semantic similarity judgments for 267,095 intra- and inter-modality pairs. We report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC’s value for measuring the influence of intra- and inter-modality learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “To alleviate this, we use an image-text dualencoder(2) [38] pretrained on real images in the Conceptual Captions dataset [48], which is disjoint from MS-COCO.”
MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “We report results on Crisscrossed Captions (CxC) (Parekh et al., 2021), which additionally provides image-text, text-text, and image-image similarity ratings.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “We also evaluate ALIGN on Crisscrossed Captions (CxC) (Parekh et al., 2021), which is an extension of MSCOCO with additional human semantic similarity judgments for caption-caption, image-image, and image-caption pairs.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “Following prior work [16, 26], we pre-train on Conceptual Captions [32] and then train on LN-COCO (the Localized Narratives portion for MS-COCO images).”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “…than 10K training examples) as target tasks.11 10Beyond the datasets from §2, we use DOCNLI (Yin et al., 2021), YELP-2 (Zhang et al., 2015), CXC (Parekh et al., 2021), DROP (Dua et al., 2019), WINOGRANDE (Sakaguchi et al., 2020), HELLASWAG (Zellers et al., 2019), COSMOSQA (Huang et al., 2019),…”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, methodology): “Following Parekh et al. (2020), we pretrain the CNN in an image-text dual encoder setting using the Conceptual Captions dataset (Sharma et al., 2018).”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “uate our best models (with BERT and Grid features on either BUTD or WSL backbones) on the Crisscrossed Captions(CxC) [35] extension of COCO, which evaluates imagetext matching systems more holistically with additional intramodal and inter-modal semantic similarity annotations.”
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2022, methodology): “Inspired by Parekh et al. (2021) and Zang et al. (2021), we also build a retrieval model R named DE++ which consists of a modality intent prediction module R α and a ranking module R β .”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Unfortunately, the Crisscrossed Captions (CxC) [165] dataset provides ratings only for the COCO evaluation splits.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “• CxC [97] is extended based on MS-COCO dataset by rating existing and new pairs with continuous (0-5) semantic similarity.”