Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
By Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond J. Mooney, Kate Saenko, Trevor Darrell. 2015
292 citations recorded by the source corpus.
While recent deep neural network models have achieved promising results on the image captioning task, they rely largely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired imagesentence datasets. Our method achieves this by leveraging large object recognition datasets and external text corpora and by transferring knowledge between semantically similar concepts. Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet. In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model's ability to describe novel concepts by empirically evaluating its performance on MSCOCO and show qualitative results on ImageNet images of objects for which no paired image-sentence data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context.
Subfields: Computer Vision and Pattern Recognition, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Andrew Zisserman
- Jitendra Malik
- Ming-Hsuan Yang
In Natural Language Generation
- Kate Saenko
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Lisa Anne Hendricks
- Marcus Rohrbach
- Matteo Stefanini
- Subhashini Venugopalan
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “This problem is distinct from the novel object captioning problem [1,23], which relies on abundant in-domain data but zero out-of-domain data.”
- Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “For example, methods have been proposed to integrate new vocabulary with limited [40] or no [68] examples of images and corresponding captions.”
- Generating Visual Explanations (2016, background): “For example, [40,41] propose introducing new vocabulary words into a captioning system.”
- Residual Attention Network for Image Classification (2017, methodology): “Sequential process [23, 12, 37, 7] models image classification as a sequential decision.”
- Dual Attention Networks for Multimodal Reasoning and Matching (2016, background): “Recently, dramatic advances in deep learning have broken the boundaries between vision and language, drawing growing interest in their intersection, such as visual question answering (VQA) [3, 37, 23, 35], image captioning [33, 2], image-text matching [8, 11, 20, 30], visual grounding [24, 9], etc.”
- Neural Baby Talk (2018, methodology): “We report F1 as in [4], which checks if the specific excluded object is mentioned in the generated caption.”
- Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, methodology): “To introduce an additional vocabulary word, the GloVe embedding for the new word is simply concatenated with W e as an additional column, increasing the dimension of both Π t and p t by one.”
- Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Most commonly, caption generators use an activation layer from the pre-trained network as their input features (e.g. Kiros et al., 2014; Karpathy et al., 2014; Karpathy & Fei-Fei, 2015; Vinyals et al., 2015; Mao et al., 2015a; Xu et al., 2015; Yagcioglu et al., 2015; Hendricks et al., 2016b).”
- VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “The early works, such as Deep Compositional Captioner (Hendricks et al. 2016) and Novel Object Captioner (Venugopalan et al. 2017), propose to use unpaired image and sentence data to transfer knowledge among semantically similar visual concepts.”
- Weakly-Supervised Learning of Visual Relations (2017, background): “Zero-shot learning has been mostly explored for object classification [12, 28, 45, 47] and recently for the task of describing images with novel objects [18, 46].”
- Semantic Compositional Networks for Visual Captioning (2016, background): “Other related work includes [32] for video captioning and [1] for composing sentences that describe novel objects.”
- Zero-Shot Object Detection (2018, methodology): “llion bounding boxes that span 545 object categories. Procedure for Creating Train and Test Splits: For dividing the classes into seen (train) and unseen (test) classes, we use a procedure similar to [57]. We begin with word-vector embeddings for all classes and cluster them into Kclusters using cos”
- Object Hallucination in Image Captioning (2018, methodology): “jects are held out, which we call the Novel Object split (Hendricks et al., 2016).”
- A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “Anne et al. [6] proposed a Deep Compositional Captioner (DCC) that can represent the unseen objects in generated captions.”
- Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning (2018, background): “A successful FGVC model should be able to discriminate categories with subtle differences, which presents formidable challenges for the model design yet also provides insights to a wide range of applications such as rich image captioning [3], image generation [5], and machine teaching [27, 37].”
- Long-Term Recurrent Convolutional Networks for Visual Recognition and Description (2017, methodology): “For example, methods have been proposed to integrate new vocabulary with limited [40] or no [68] examples of images”
- Learning to Learn Words from Visual Scenes (2019, background): “Finally, unlike [4], our approach does not require any side information or external information, and instead acquires new words using their surrounding textual and visual context.”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “[176] introduced a variant of the COCOdataset [128], called held-out COCO, inwhich image-caption pairs containing one of eight pre-selected object classes were removed from the training set but not from the test set.”