Hierarchy Parsing for Image Captioning

By Ting Yao, Yingwei Pan, Yehao Li, Tao Mei. 2019

183 citations recorded by the source corpus.

It is always well believed that parsing an image into constituent visual patterns would be helpful for understanding and representing an image. Nevertheless, there has not been evidence in support of the idea on describing an image with a natural-language utterance. In this paper, we introduce a new design to model a hierarchy from instance level (segmentation), region level (detection) to the whole image to delve into a thorough image understanding for captioning. Specifically, we present a HIerarchy Parsing (HIP) architecture that novelly integrates hierarchical structure into image encoder. Technically, an image decomposes into a set of regions and some of the regions are resolved into finer ones. Each region then regresses to an instance, i.e., foreground of the region. Such process naturally builds a hierarchal tree. A tree-structured Long Short-Term Memory (Tree-LSTM) network is then employed to interpret the hierarchal structure and enhance all the instance-level, region-level and image-level features. Our HIP is appealing in view that it is pluggable to any neural captioning models. Extensive experiments on COCO image captioning dataset demonstrate the superiority of HIP. More remarkably, HIP plus a top-down attention-based LSTM decoder increases CIDEr-D performance from 120.1% to 127.2% on COCO Karpathy test split. When further endowing instance-level and region-level features from HIP with semantic relation learnt through Graph Convolutional Networks (GCN), CIDEr-D is boosted up to 130.6%.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Luc Van Gool
3. Rita Cucchiara
4. Dacheng Tao
5. Yi Yang
6. Zicheng Liu
7. Lorenzo Baraldi
8. Marcella Cornia

In Natural Language Processing

1. Rita Cucchiara
2. Lorenzo Baraldi
3. Marcella Cornia
4. Hanwang Zhang
5. Xu Yang
6. Matteo Stefanini
7. Jianfei Cai
8. Kai Yi

In Multimodal Learning

1. Rita Cucchiara
2. Lorenzo Baraldi
3. Marcella Cornia
4. Zhe Gan
5. Zicheng Liu
6. Lijuan Wang
7. Giuseppe Fiameni
8. Xiaowei Hu

Citations in the corpus

• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Many improvements to the encoder [11,38,49,75,76,80,81], the decoder [72, 73, 78], and the attention mechanism [8, 13, 24, 33, 36] has since been proposed.”
• Meshed-Memory Transformer for Image Captioning (2019, neutral): “6 GCN-LSTM+HIP [49] - 39.”
• From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “Because of their representation power, LSTMs with two-layers and internal attention mechanisms represent the most employed language model approach before the advent of Transformer-based architectures [68], [71], [72], [73].”
• Dual-Level Collaborative Transformer for Image Captioning (2021, unknown): “The compared models include: SCST (Rennie et al. 2017), Up-Down (Anderson et al. 2018), HAN (Wang, Chen, and Hu 2019), GCN-LSTM (Yao et al. 2018), SGAE (Yang et al. 2019), ORT (Herdade et al. 2019), SRT (Wang et al. 2020), AoA (Huang et al. 2019), HIP (Yao et al. 2019), M2 (Cornia et al. 2020) X-Tra”
• Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, background): “Image captioning [3, 20, 17, 7, 51] aims to generate a natural language sentence to describe the image content.”
• Deconfounded Image Captioning: A Causal Retrospect (2020, background): “Many studies fall into this large-scale training strategy, e.g., Up-Down [24] exploited Visual Genome [32] with a larger label space, HIP [33] used more fine-grained segmentation annotations, and vision-language BERT frameworks [34], [35] trained their network by 3 million image-caption pairs from…”
• Injecting Semantic Concepts into End-to-End Image Captioning (2021, methodology): “Despite these significant advances, most of the main-stream captioning models [11,53,63,81] rely heavily on a bulky object detector to provide regional visual representations for the multimodal interaction, as shown in Figure 1-a.”
• X-Linear Attention Networks for Image Captioning (2020, background): “Image captioning is an active research area [2, 12, 19, 23, 24, 28, 33, 34, 35, 37, 39, 40, 41].”