Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects

By Ting Yao, Yingwei Pan, Yehao Li, Tao Mei. 2017

154 citations recorded by the source corpus.

Image captioning often requires a large set of training image-sentence pairs. In practice, however, acquiring sufficient training pairs is always expensive, making the recent captioning models limited in their ability to describe objects outside of training corpora (i.e., novel objects). In this paper, we present Long Short-Term Memory with Copying Mechanism (LSTM-C) — a new architecture that incorporates copying into the Convolutional Neural Networks (CNN) plus Recurrent Neural Networks (RNN) image captioning framework, for describing novel objects in captions. Specifically, freely available object recognition datasets are leveraged to develop classifiers for novel objects. Our LSTM-C then nicely integrates the standard word-by-word sentence generation by a decoder RNN with copying mechanism which may instead select words from novel objects at proper places in the output sentence. Extensive experiments are conducted on both MSCOCO image captioning and ImageNet datasets, demonstrating the ability of our proposed LSTM-C architecture to describe novel objects. Furthermore, superior results are reported when compared to state-of-the-art deep models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “…our approach works more robustly across in-and out-of-domain scenarios and is simpler than the state-of-the-art techniques that utilize constrained beam search (CBS) [5], ﬁnite state machine construction plus CBS [6], generating slot-ﬁlling templates [54, 76], and copying mechanisms [78].”
nocaps: novel object captioning at scale (2019, methodology): “Related to Neural Baby Talk, the LSTM-C [47] model augments a standard recurrent neural network sentence decoder with a copying mechanism which may select words corresponding to object detector predictions to appear in the output sentence.”
Neural Baby Talk (2018, methodology): “Note that we do not pre-train the language model using COCO captions as in [4, 42, 48], and simply replace the novel object’s word embedding with an existing one which belongs to the same super-category in COCO (e.”
Exploring Visual Relationship for Image Captioning (2018, methodology): “ not yet fully understood. 2 Related Work Image Captioning. With the prevalence of deep learning [17] in computer vision, the dominant paradigm in modern image captioning is sequence learning methods [7,34,37,38,39,40] which utilize CNN plus RNN model to generate novel sentences with exible syntacti”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “Yao et al. (2017) use LSTM-C with a copying mechanism to assemble the detected novel objects for caption generation.”
Boosting Image Captioning with Attributes (2016, background): “The research on image captioning has proceeded along three different dimensions: template-based methods [11, 18, 34], search-based approaches [4, 7], and languagebased models [5, 16, 31, 32, 33, 36, 38].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “Yao et al. [154] proposed a copying mechanism to generate description for novel objects.”
Decoupled Novel Object Captioner (2018, unknown): “Therefore, to better evaluate the description quality, we also use the F1-score as an evaluation metric following [3, 34, 41].”
X-Linear Attention Networks for Image Captioning (2020, background): “Image captioning is an active research area [2, 12, 19, 23, 24, 28, 33, 34, 35, 37, 39, 40, 41].”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Some approaches to this variant [179], [180] integrate copying mechanisms in the languagemodel to select novel objects predicted from a tagger or”
MetaSearch: Incremental Product Search via Deep Meta-Learning (2020, background): “In recent years, many studies have focused on product image classiﬁcation and recognition [28], [53], [54].”
Pointing Novel Objects in Image Captioning (2019, unknown): “By augmenting the standard RNN-based language model with the object/concept learners, LSTM-C leads to a performance boost against NOC that produces novel objects purely depending on generative mechanism in LSTM.”
Hierarchy Parsing for Image Captioning (2019, background): “In addition, image captioning could be extended to novel object captioning [17, 33] which leverages unpaired image/text data to describe novel objects or image paragraph generation [30] which produces a coherent paragraph to depict an image.”
Learning to Learn Words from Visual Scenes (2019, background): “Previous work on captioning [28,31,62,60] produces words never seen in the ground truth captions.”
LSDSCC: a Large Scale Domain-Specific Conversational Corpus for Response Generation with Diversity Oriented Evaluation Metrics (2018, background): “This phenomenon indicates a relationship between relevance and diversity against that in some of text generation tasks (e.g., image caption (Yao et al., 2017)).”
Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, background): “Traditionally, this has been addressed through the Novel Object Captioning protocol [1,2,64], which requires the captioner to name objects unseen in training captions, relying on object detection tags [20] or uni-modal pre-training [18].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “, 2017a) and LSTM-C (Yao et al., 2017) address the challenge of generating description for these images.”
Partially-Supervised Image Captioning (2018, methodology): “More recently, approaches based on constrained beam search [13], word copying [33] and (2)www.”