Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach

By Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In-So Kweon. 2019

61 citations recorded by the source corpus.

Constructing an organized dataset comprised of a large number of images and several captions for each image is a laborious task, which requires vast human effort. On the other hand, collecting a large number of images and sentences separately may be immensely easier. In this paper, we develop a novel data-efficient semi-supervised framework for training an image captioning model. We leverage massive unpaired image and caption data by learning to associate them. To this end, our proposed semi-supervised learning method assigns pseudo-labels to unpaired samples via Generative Adversarial Networks to learn the joint distribution of image and caption. To evaluate, we construct scarcely-paired COCO dataset, a modified version of MS COCO caption dataset. The empirical results show the effectiveness of our method compared to several strong baselines, especially when the amount of the paired samples are scarce.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Kim et al [29] focus on learning efficiency and improve the data efficiency by learning from auxiliary unpaired image-caption data.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Moreover, semisupervised approaches have been proposed, such as [189], which uses both paired and unpaired data with adversarial learning, and [190], which performs iterative self-learning.”
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision (2021, background): “Also, related is the work [20] which utilizes both paired and unpaired data for the task of image captioning.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, result): “In contrast, prior works on semi-supervised cross-modal learning such as image captioning [6, 21] are more relevant.”
From Show to Tell: A Survey on Image Captioning (2021, background): “Some alternatives exist, such as [163] that generates a caption from the image scene-graph, [164] that leverages a memory-based network, or [165], [166] that propose semi-supervised strategies.”
Intra-agent speech permits zero-shot task acquisition (2022, background): “Our work’s analysis of the scaling curve [31] properties of semi-supervised captioning [50–52] may provide additional analytical tools for the field.”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, unknown): “Semi-supervised methods are also available [35].”