Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs

By Shizhe Chen, Qin Jin, Peng Wang, Qi Wu. 2020

250 citations recorded by the source corpus.

Humans are able to describe image contents with coarse to fine details as they wish. However, most image captioning models are intention-agnostic which cannot generate diverse descriptions according to different user intentions initiatively. In this work, we propose the Abstract Scene Graph (ASG) structure to represent user intention in fine-grained level and control what and how detailed the generated description should be. The ASG is a directed graph consisting of three types of abstract nodes (object, attribute, relationship) grounded in the image without any concrete semantic labels. Thus it is easy to obtain either manually or automatically. From the ASG, we propose a novel ASG2Caption model, which is able to recognise user intentions and semantics in the graph, and therefore generate desired captions following the graph structure. Our model achieves better controllability conditioning on ASGs than carefully designed baselines on both VisualGenome and MSCOCO datasets. It also significantly improves the caption diversity via automatically sampling diverse ASGs as control signals. Code will be released at \url{https://github.com/cshizhe/asg2cap}.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “Given key entities or arguments as keywords and fully masked contexts, we envision T ailor can help generate arguments (Schiller et al., 2021), compo-sitionally augment data (Akyürek et al., 2021), or generate captions (Chen et al., 2020).”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, neutral): “[9, 12] investigate fine-grained control of caption generation.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “…between objects that is most of interest for the user, as done for example in [246] via verbs and semantic roles (verbs represent activities in the image and semantic roles determine how objects engage in these activities) and in [247], [248] via user-generated or user-selected scene graphs.”
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning (2021, background): “Image captioning recently shifted into new directions such as generating diverse descriptions [38, 12, 44] by allowing both grounding and controllability [9, 47, 7] while using various contextual information [6, 35].”
Understanding Guided Image Captioning Performance across Domains (2020, methodology): “2017), to personalized image captioning which changes the style of captions (Chunseong Park, Kim, and Kim 2017) or to produce a caption using multiple entities from the image (You et al. 2016; Chen et al. 2020).”
Deconfounded Image Captioning: A Causal Retrospect (2020, methodology): “Inspired by this, researchers [44], [45], [46] proposed to learn a scene graph from the image and to apply Graph Neural Network [47], [48], [49] to transform this Fig.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “proposed an ASG (Abstract Scene Graph) structure [163] to represent”