Captioning Images with Diverse Objects

By Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond J. Mooney, Trevor Darrell, Kate Saenko. 2016

186 citations recorded by the source corpus.

Recent captioning models are limited in their ability to scale and describe concepts unseen in paired image-text corpora. We propose the Novel Object Captioner (NOC), a deep visual semantic captioning model that can describe a large number of object categories not present in existing image-caption datasets. Our model takes advantage of external sources – labeled images from object recognition datasets, and semantic knowledge extracted from unannotated text. We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets. We demonstrate that our model exploits semantic information to generate captions for hundreds of object categories in the ImageNet object recognition dataset that are not observed in MSCOCO image-caption training data, as well as many categories that are observed very rarely. Both automatic evaluations and human judgements show that our model considerably outperforms prior work in being able to describe many more categories of objects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Novel object captioning aims to describe objects that do not exist in the training data [1, 23, 41, 50, 70].”
nocaps: novel object captioning at scale (2019, methodology): “To obtain a captioning model that performs well in the target domain, the Deep Compositional Captioner [13] and its extension, the Novel Object Captioner [40], both attempt to transfer knowledge by leveraging object detection datasets and external text corpora by decomposing the captioning model int”
Neural Baby Talk (2018, methodology): “Note that we do not pre-train the language model using COCO captions as in [4, 42, 48], and simply replace the novel object’s word embedding with an existing one which belongs to the same super-category in COCO (e.”
Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, methodology): “Unsur-Input image containing previously unseen object (‘suitcase’)”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “The early works, such as Deep Compositional Captioner (Hendricks et al. 2016) and Novel Object Captioner (Venugopalan et al. 2017), propose to use unpaired image and sentence data to transfer knowledge among semantically similar visual concepts.”
Weakly-Supervised Learning of Visual Relations (2017, background): “Zero-shot learning has been mostly explored for object classiﬁcation [12, 28, 45, 47] and recently for the task of describing images with novel objects [18, 46].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “ recognize the unseen objects from unpaired images and incorporate them with LSTM to generate captions. Generating captions for the unseen images is a challenging research problem. Venugopalan et al. [140] introduced a Novel Object Captioner (NOC) for generating captions for unseen objects in the im”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, background): “Image and video captioning [1, 6, 9, 11, 16, 18, 21, 35, 38, 39, 40] is the task of generating a textual description of an image, or similarly, ﬁnding the textual description that best matches the image.”
Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks (2018, methodology): “For example, many NLP (Zou et al., 2013; Seo et al., 2017; Lee et al., 2017; Chen et al., 2017) and computer vision applications (Venugopalan et al., 2017; Teney et al., 2017) have successfully utilized word embeddings trained on a large corpus (Mikolov et al., 2013; Pennington et al., 2014).”
Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models (2020, unknown): “…any image in the form of a sentence is also well known as image captioning and has been at the forefront of research for both computer vision and natural language processing (Vinyals et al., 2014; Karpathy & Feifei, 2014; Venugopalan et al., 2016; Selvaraju et al., 2019; Vedantam et al., 2017).”
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (2019, background): “Another set of closely related work utilized additional data for scaling up either vision tasks (Hoffman et al., 2016; Tang et al., 2017; Redmon and Farhadi, 2017) or vision and language tasks (Venugopalan et al., 2017; Lu et al., 2018; Noh et al., 2019).”
Pointing Novel Objects in Image Captioning (2019, unknown): “[26] further extend DCC by simultaneously optimizing the visual recognition network, language model, and image captioning network in an end-to-end manner.”
Decoupled Novel Object Captioner (2018, unknown): “tion datasets and a pre-trained language sequence model from external text corpora. The captioning model is trained on the paired image-sentence data with above pre-trained models. Venugopalan et al. [34] discussed a Novel Object Captioner (NOC) to further improve the DCC to an end-to-end system by ”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “, 2016), Novel object Captioner (NOC) (Venugopalan et al., 2017), Constrained Beam Search (CBS) (Anderson et al.”
Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases (2020, methodology): “ie the input and output embeddings using W in Seq2Seq terminology. In this way, we can assign semantic meanings to words from pre-trained embeddings, which can facilitate the open vocabulary learning [41]. 4.2 BERT Encoding Now we discuss how we use BERT to compute W and � (�,V,�). BERT has demonstr”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, background): “Novel Object Captioning in Images Recent studies on novel object captioning (Anne Hendricks et al. 2016; Venugopalan et al. 2017) attempt to describe novel objects not appearing during training.”
From Show to Tell: A Survey on Image Captioning (2021, background): “Early approaches to this task [151], [152] tried to transfer knowledge from out-domain images by conditioning the model on external unpaired visual and textual data at training time.”
Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects (2017, methodology): “Please note that for fair comparison with LRCN and DCC which utilize one hot vector as their word representations, we include two runs, i.e., NOC (One hot) and NOC (One hot+Glove) which are our implementations of NOC.”