Deep Visual-Semantic Alignments for Generating Image Descriptions
By Andrej Karpathy, Li Fei-Fei. 2014
6,086 citations recorded by the source corpus.
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks (RNN) over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions outperform retrieval baselines on both full images and on a new dataset of region-level annotations. Finally, we conduct large-scale analysis of our RNN language model on the Visual Genome dataset of 4.1 million captions and highlight the differences between image and region-level caption statistics.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Percy Liang
- Quoc V. Le
- Luke S. Zettlemoyer
- Trevor Darrell
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Richard Socher
- Yejin Choi
- Graham Neubig
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Caiming Xiong
In Multimodal Learning
Citations in the corpus
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “We use the same splits as in (Karpathy and Fei-Fei, 2015), i.e., we use 113k images from the COCO dataset (each containing 5 captions) for training, 5k images for validation and 5k images for test.”
- Skip-Thought Vectors (2015, methodology): “For development and testing we use the same splits as [ 31 ] .”
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “The field has tackled a diverse set of tasks such as captioning [15, 13, 40, 39, 24], alignment [13, 17, 38], Q&A [22, 21], visual model learning from textual descriptions [9, 29], and semantic visual search with natural multisentence queries [19].”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “We follow Karpathy & Fei-Fei (2015) to split the data into 30K/1K/1K training/validation/test splits.”
- SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, methodology): “We use the same splits as Karpathy and Fei-Fei (2015), i.e., we use 113k images (each containing 5 captions) for training, 5k images for validation and 5k images for test.”
- What makes ImageNet good for transfer learning? (2016, methodology): “Using ImageNet pre-trained CNN features, impressive results have been obtained on several image classification datasets [13, 35], object detection [16, 39], action recognition [41], human pose estimation [9], image segmentation [10], optical flow [47], image captioning [12, 22] and others [26].”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “For all tasks, we use the Karpathy train split (Karpathy & Fei-Fei, 2015) of COCO for task-agnostic pre-training, which has around 100k images with 5 captions each.”
- VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [11, 8, 15]), which is a model of the form p(y|x), where y is the manually provided caption and x is the image.”
- VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Compared to basic tokenization schemes often used for image captioning that split on whitespace [10, 11], BPE makes fewer linguistic assumptions, exploits subword information, and results in fewer out-of-vocab tokens.”
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “1K Test Set 5K Test Set DVSA [14] - 38.”
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “For example, Text-Image Retrieval[4] aims to retrieve the most relevant image given a text, or vice versa.”
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “For downstream tasks, such as image classi-fication, image retrieval (Karpathy and Fei-Fei 2015) (Lee et al. 2018) and object detection (Ren et al. 2015), the resulting models can extract feature representations for input images, which will be further used in following task-specific models.”
- Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models (2019, methodology): “…−→ people Plural form of boy/girl/guy/gal −→ youngsters Gendered pronouns (his/her/hers/him/himself/herself) −→ it/itself/its Note that for all our experiments, we use the train/val/test splits described in the work [Karpathy and Fei-Fei, 2015], also used for the CNN model [Aneja et al., 2018].”
- Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “Following [23], we use the training, testing and validation split.”
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, background): “Automatic image captioning has a long history (Hodosh et al., 2013; Donahue et al., 2014; Karpathy and Fei-Fei, 2015; Kiros et al., 2015).”
- ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “Moreover, multimodal tasks like image captioning (Karpathy and Li, 2015) or visual question answering (VQA) (Antol et al.”