Pointing Novel Objects in Image Captioning
By Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, Tao Mei. 2019
74 citations recorded by the source corpus.
Image captioning has received significant attention with remarkable improvements in recent advances. Nevertheless, images in the wild encapsulate rich knowledge and cannot be sufficiently described with models built on image-caption pairs containing only in-domain objects. In this paper, we propose to address the problem by augmenting standard deep captioning architectures with object learners. Specifically, we present Long Short-Term Memory with Pointing (LSTM-P) --- a new architecture that facilitates vocabulary expansion and produces novel objects via pointing mechanism. Technically, object learners are initially pre-trained on available object recognition data. Pointing in LSTM-P then balances the probability between generating a word through LSTM and copying a word from the recognized objects at each time step in decoder stage. Furthermore, our captioning encourages global coverage of objects in the sentence. Extensive experiments are conducted on both held-out COCO image captioning and ImageNet datasets for describing novel objects, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, we obtain an average of 60.9% in F1 score on held-out COCO dataset.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Giuseppe Fiameni
- Rita Cucchiara
- Marcella Cornia
- Lorenzo Baraldi
- Tao Mei
- Ting Yao
- Yingwei Pan
- Yehao Li
In Computer Vision and Pattern Recognition
- Tao Mei
- Rita Cucchiara
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Lorenzo Baraldi
- Marcella Cornia
- Trevor Darrell
In Multimodal Learning
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Tao Mei
- Ting Yao
- Matteo Stefanini
- Yingwei Pan
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Marimuthu Kalimuthu
- Dietrich Klakow
- Aditya Mogadala
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Novel object captioning aims to describe objects that do not exist in the training data [1, 23, 41, 50, 70].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Some approaches to this variant [179], [180] integrate copying mechanisms in the languagemodel to select novel objects predicted from a tagger or”
- ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, unknown): “, pre-training, training, or inference) [3, 18, 31, 42, 46].”
- Learning to Learn Words from Visual Scenes (2019, background): “Previous work on captioning [28,31,62,60] produces words never seen in the ground truth captions.”
- X-Linear Attention Networks for Image Captioning (2020, background): “Image captioning is an active research area [2, 12, 19, 23, 24, 28, 33, 34, 35, 37, 39, 40, 41].”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “While earlier research addresses only natural images, (Li et al., 2019) incorporated medical domain knowledge to generate realistic and accurate descriptions for medical images.”
- Hierarchy Parsing for Image Captioning (2019, background): “In addition, image captioning could be extended to novel object captioning [17, 33] which leverages unpaired image/text data to describe novel objects or image paragraph generation [30] which produces a coherent paragraph to depict an image.”