Unsupervised Image Captioning

By Yang Feng, Lin Ma, Jiebo Luo. 2018

226 citations recorded by the source corpus.

Deep neural networks have achieved great successes on the image captioning task. However, most of the existing models depend heavily on paired image-sentence datasets, which are very expensive to acquire. In this paper, we make the first attempt to train an image captioning model in an unsupervised manner. Instead of relying on manually labeled image-sentence pairs, our proposed model merely requires an image set, a sentence corpus, and an existing visual concept detector. The sentence corpus is used to teach the captioning model how to generate plausible sentences. Meanwhile, the knowledge in the visual concept detector is distilled into the captioning model to guide the model to recognize the visual concepts in an image. In order to further encourage the generated captions to be semantically consistent with the image, the image and caption are projected into a common latent space so that they can reconstruct each other. Given that the existing sentence corpora are mainly designed for linguistic research and are thus with little reference to image contents, we crawl a large-scale image description corpus of two million natural sentences to facilitate the unsupervised image captioning scenario. Experimental results show that our proposed model is able to produce quite promising results without any caption annotations.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Feng et al. [20] propose unsupervised captioning without using paired image-caption supervision.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “adversarial learning by training anLSTM-baseddiscriminator to distinguish whether a caption is real or generated [185], [186].”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, background): “Unsupervised approaches can be achieved by training with a visual concept detector or by learning a joint image-language embedding space [19, 40].”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, unknown): “[37] 2019 CNN/InceptionV4, RNN/LSTM Image, Text MS-COCO BLEU, ROUGE, CIDEr, METEOR, SPICE”
Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach (2019, methodology): “We also compare the recent unpaired image captioning methods [Gu et al. , 2018; Feng et al. , 2019] in Table 1.”
Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models (2020, unknown): “There have been very few works which have used unsupervised (Feng et al., 2018) or reinforcement learning (Ren et al., 2017) for captioning images.”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, background): “However, what makes video captioning more challenging than image captioning [12, 44, 7, 17] is not only that the input of video captioning are multiple images, but also that video contains richer semantics, such as spatial/temporal information, content/motion information, and even speech information”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “[94], ClipCap [51], Oscar [40], and LEMON [26]; and (2) weakly supervised methods: UIC [19], IC-SME [37], S2S-SS and S2S-GCC [25].”
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021, methodology): “, 2018) and unsupervised image captioning (Feng et al., 2019).”
From Show to Tell: A Survey on Image Captioning (2021, background): “After this work, the most common approach focuses on adversarial learning by training an LSTM-based discriminator to distinguish whether a caption is real or generated [161], [162].”
Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, background): “Lots of methods have attempted to address the multi-modal learning problems through dealing with visual-semantic learning tasks, such as visual question answering [32, 56, 54] and image captioning [10, 49, 12].”