Connecting Vision and Language with Localized Narratives

By Jordi Pont-Tuset, Jasper R. R. Uijlings, Soravit Changpinyo, Radu Soricut, Vittorio Ferrari. 2019

310 citations recorded by the source corpus.

We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “We consider a suite of datasets that are commonly used in literature [53, 54, 56, 59], including MSCOCO [4], CUB [47], LN-COCO [36], Multi-modal CelebA-HQ (MM CelebA-HQ) [52].”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, background): “Localized Narratives [40] contains much longer descriptions, which increases the difficulty of textto-image synthesis (see Sec.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “…down-stream generalization capabilities [6, 15, 37, 60], prompted us to construct a large-scale pretraining dataset from various publicly available image-text datasets, including a ﬁltered version of LAION [43], Wikipedia Image-Text [48], CC3M [8], Visual Genome [26] and Localized Narratives [35].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “In the Supplementary material, we also explore using the Open Images Localized Narratives dataset (LocNar) [61], as an alternative “in-domain” (from a visual standpoint) pre-training data source.”
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “We also experiment with the Localized Narratives dataset (Pont-Tuset et al., 2019).”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “[55] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “While most recent work has focused exclusively on the MS-COCO benchmark, we also show that strong zero-shot and finetuned results can be achieved on the Localized Narratives dataset [29], which has descriptions that are four times longer than MS-COCO’s on average.”
Text-to-Image Generation Grounded by Fine-Grained User Attention (2020, methodology): “Localized Narratives [28] is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases.”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, methodology): “As with Localized Narratives, RxR Guides transcribe their own recordings; this produces high quality text versions of the instructions.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “[244] and user-provided visual words [220], or dense, as mouse traces [138], [245].”
Lafite2: Few-shot Text-to-Image Generation (2022, methodology): “Standard datasets are considered for downstream tasks, including MS-COCO [19], CUB [35], MM-CelebA-HQ [38], LN-COCO [24].”
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation (2021, methodology): “We mainly use Open Images [18] dataset with localized narratives and synchronized speech that are provided by [29] as our pre-training dataset.”
Pre-training image-language transformers for open-vocabulary tasks (2022, methodology): “To allow image-text pre-training, the Localized Narratives dataset [9], which provides additional caption annotations for OpenImages, is used (only its OpenImages subset).”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, unknown): “…Captions (CC) [63], SBU Captions [52], Flickr30k [88], VQA 2.0 [27], VCR [89], NLVR2 [74], SNLI-VE [87], QNLI [57], MLNI-mm [84], QQP [36], Localized Narratives (LN) [55], Wikipedia Image Text (WIT) [73], Conceptual Captions 12M (CC 12M) [10], Red Caps (RC) [15], YFCC100M [77], and SST-2 [72].”
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, background): “Here, we test the performance of our model by replacing our self-collected dataset in the second-stage with the Localized Narratives dataset.”