Partially-Supervised Image Captioning

By Peter Anderson, Stephen Gould, Mark Johnson. 2018

34 citations recorded by the source corpus.

Image captioning models are becoming increasingly successful at describing the content of images in restricted domains. However, if these models are to function in the wild - for example, as assistants for people with impaired vision - a much larger number and variety of visual concepts must be understood. To address this problem, we teach image captioning models new visual concepts from labeled images and object detection datasets. Since image labels and object classes can be interpreted as partial captions, we formulate this problem as learning from partially-specified sequence data. We then propose a novel algorithm for training sequence models, such as recurrent neural networks, on partially-specified sequences which we represent using finite state automata. In the context of image captioning, our method lifts the restriction that previously required image captioning models to be trained on paired image-sentence corpora only, or otherwise required specialized model architectures to take advantage of alternative data modalities. Applying our approach to an existing neural captioning model, we achieve state of the art results on the novel object captioning task using the COCO dataset. We further show that we can train a captioning model to describe new visual concepts from the Open Images dataset while maintaining competitive COCO evaluation scores.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Trevor Darrell
3. Dacheng Tao
4. Devi Parikh
5. Xiaogang Wang
6. Cordelia Schmid
7. Ming-Hsuan Yang
8. Bernt Schiele

In Multimodal Learning

1. Devi Parikh
2. Dhruv Batra
3. Radu Soricut
4. Mohit Bansal
5. Zhe Gan
6. Lijuan Wang
7. Stefan Lee
8. Soravit Changpinyo

In Natural Language Generation

1. Lin Ma
2. Devi Parikh
3. Dhruv Batra
4. Jiebo Luo
5. Yang Feng
6. Mark Johnson
7. Radu Soricut
8. Nigel Collier

Citations in the corpus

• Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “…our approach works more robustly across in-and out-of-domain scenarios and is simpler than the state-of-the-art techniques that utilize constrained beam search (CBS) [5], finite state machine construction plus CBS [6], generating slot-filling templates [54, 76], and copying mechanisms [78].”
• nocaps: novel object captioning at scale (2019, methodology): “Such objects which have detection annotations but are not present in caption corpora are referred to as novel objects and the task of describing images containing novel objects is termed novel object captioning [2,3,13,25,40,43,47].”
• Unsupervised Image Captioning (2018, methodology): “[2] trained an image caption model with partial supervision.”
• Language Models Can See: Plugging Visual Controls in Text Generation (2022, background): “• [2] : Choose this score when you are hesitant between the score 1 and score 3.”