Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions

By Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara. 2018

199 citations recorded by the source corpus.

Current captioning approaches can describe images using black-box architectures whose behavior is hardly controllable and explainable from the exterior. As an image can be described in infinite ways depending on the goal and the context at hand, a higher degree of controllability is needed to apply captioning algorithms in complex scenarios. In this paper, we introduce a novel framework for image captioning which can generate diverse descriptions by allowing both grounding and controllability. Given a control signal in the form of a sequence or set of image regions, we generate the corresponding caption through a recurrent architecture which predicts textual chunks explicitly grounded on regions, following the constraints of the given control. Experiments are conducted on Flickr30k Entities and on COCO Entities, an extended version of COCO in which we add grounding annotations collected in a semi-automatic manner. Results demonstrate that our method achieves state of the art performances on controllable image captioning, in terms of caption quality and diversity. Code and annotations are publicly available at: https://github.com/aimagelab/show-control-and-tell.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Yi Yang
  4. Trevor Darrell
  5. Chunhua Shen
  6. Xiaogang Wang
  7. Ming-Hsuan Yang
  8. Jitendra Malik

In Multimodal Learning

  1. Rita Cucchiara
  2. Lorenzo Baraldi
  3. Marcella Cornia
  4. Radu Soricut
  5. Giuseppe Fiameni
  6. Matteo Stefanini
  7. Soravit Changpinyo
  8. Piyush Sharma

In Natural Language Generation

  1. Rita Cucchiara
  2. Lorenzo Baraldi
  3. Marcella Cornia
  4. Giuseppe Fiameni
  5. Radu Soricut
  6. Matteo Stefanini
  7. Marimuthu Kalimuthu
  8. Dietrich Klakow

Citations in the corpus