Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
By Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara. 2018
199 citations recorded by the source corpus.
Current captioning approaches can describe images using black-box architectures whose behavior is hardly controllable and explainable from the exterior. As an image can be described in infinite ways depending on the goal and the context at hand, a higher degree of controllability is needed to apply captioning algorithms in complex scenarios. In this paper, we introduce a novel framework for image captioning which can generate diverse descriptions by allowing both grounding and controllability. Given a control signal in the form of a sequence or set of image regions, we generate the corresponding caption through a recurrent architecture which predicts textual chunks explicitly grounded on regions, following the constraints of the given control. Experiments are conducted on Flickr30k Entities and on COCO Entities, an extended version of COCO in which we add grounding annotations collected in a semi-automatic manner. Results demonstrate that our method achieves state of the art performances on controllable image captioning, in terms of caption quality and diversity. Code and annotations are publicly available at: https://github.com/aimagelab/show-control-and-tell.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Yi Yang
- Trevor Darrell
- Chunhua Shen
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
In Multimodal Learning
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Radu Soricut
- Giuseppe Fiameni
- Matteo Stefanini
- Soravit Changpinyo
- Piyush Sharma
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Radu Soricut
- Matteo Stefanini
- Marimuthu Kalimuthu
- Dietrich Klakow
Citations in the corpus
- VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, neutral): “[9, 12] investigate fine-grained control of caption generation.”
- SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “The ability to describe images and videos is one of the core challenges in this domain and a crucial achievement towards machine intelligence [6], [7], [8], [9].”
- Connecting Vision and Language with Localized Narratives (2019, methodology): “Controlled captioning was first proposed in [13] and enables a user to specify which parts of the image they want to be described, and in which order.”
- Meshed-Memory Transformer for Image Captioning (2019, methodology): “This has usually been tackled via Recurrent Neural Network models [42, 17, 45, 44, 7], in which the sequential nature of language is modeled with the recurrent relations of either RNNs or LSTMs. Additive attention or graph-like structures [48] are often added to the recurrence [45, 14] in order to…”
- Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, methodology): “Further, we employ a coverage-based metric inspired from [7, 8] between ground-truth detections or class annotations and the predicted caption.”
- Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, unknown): “ly, many attempts to merge the two worlds have been explored, and state-of-the-art results have been obtained in many of these tasks, including visual question answering [2, 12, 42], image captioning [6, 14, 40, 49] and image-text matching [5, 10, 22, 33]. In this work, we deal with the fine-grained”
- Towards Accurate Text-based Image Captioning with Content Diversity Exploration (2021, methodology): “Signal-based methods [7, 8, 9, 11] sample different predictions based on the control signals to obtain diverse image captions.”
- Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “the descriptive caption for a given image [28], [29].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “The signal can be sparse, as selected image regions [163],”
- Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning (2021, background): “Image captioning recently shifted into new directions such as generating diverse descriptions [38, 12, 44] by allowing both grounding and controllability [9, 47, 7] while using various contextual information [6, 35].”
- Understanding Guided Image Captioning Performance across Domains (2020, methodology): “encoder-decoder methods (Lu et al., 2018; Cornia et al., 2019; Anderson et al., 2017; You et al., 2016; Chen et al., 2020; Mun et al., 2017; Xu et al., 2015; Fang et al., 2015), including the method proposed in this work, can generate diverse captions with image grounding.”
- ContCap: A scalable framework for continual image captioning (2019, background): “Since the introduction of the Neural Image Caption Generator (NIC) [4], various powerful techniques have been proposed to enhance image captioning [5], [6], [7].”
- RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, methodology): “The main development of image captioning [41, 40, 5, 14, 24] can be divided into two stages: traditional method stage and deep learning method stage.”
- Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs (2020, background): “…of image descriptions such as factual, romantic, humorous styles etc ., while the other type aims to control the description contents such as different image regions [17], objects [7, 50], and part-of-speech tags [9], so that the model is able to describe user interested contents in the image.”
- ContCap: A comprehensive framework for continual image captioning (2019, background): “From the introduction of Neural Image Caption Generator (NIC) [29], various advanced techniques have been proposed to ameliorate image captioning [19][4][1].”
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “(Cornia et al., 2018) used either a sequence or a set of image regions.”
- CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation (2023, background): “However, the loose modal interaction by cosine similarity also limits the multi-modal fusion ability [4, 10, 70].”