Learning to Collocate Neural Modules for Image Captioning

By Xu Yang, Hanwang Zhang, Jianfei Cai. 2019

84 citations recorded by the source corpus.

We do not speak word by word from scratch; our brain quickly structures a pattern like \textsc{sth do sth at someplace} and then fill in the detailed description. To render existing encoder-decoder image captioners such human-like reasoning, we propose a novel framework: learning to Collocate Neural Modules (CNM), to generate the ``inner pattern'' connecting visual encoder and language decoder. Unlike the widely-used neural module networks in visual Q\&A, where the language (\ie, question) is fully observable, CNM for captioning is more challenging as the language is being generated and thus is partially observable. To this end, we make the following technical contributions for CNM training: 1) compact module design --- one for function words and three for visual content words (\eg, noun, adjective, and verb), 2) soft module fusion and multi-step module execution, robustifying the visual reasoning in partial observation, 3) a linguistic loss for module controller being faithful to part-of-speech collocations (\eg, adjective is before noun). Extensive experiments on the challenging MS-COCO image captioning benchmark validate the effectiveness of our CNM image captioner. In particular, CNM achieves a new state-of-the-art 127.9 CIDEr-D on Karpathy split and a single-model 126.0 c40 on the official server. CNM is also robust to few training samples, \eg, by training only one sentence per image, CNM can halve the performance loss compared to a strong baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanwang Zhang
2. Giuseppe Fiameni
3. Xiansheng Hua
4. Marcella Cornia
5. Lorenzo Baraldi
6. Rita Cucchiara
7. Kaihua Tang
8. Silvia Cascianelli

In Computer Vision and Pattern Recognition

1. Hanwang Zhang
2. Rita Cucchiara
3. Marcella Cornia
4. Lorenzo Baraldi
5. Giuseppe Fiameni
6. Matteo Stefanini
7. Xiansheng Hua
8. Silvia Cascianelli

In Multimodal Learning

1. Rita Cucchiara
2. Lorenzo Baraldi
3. Marcella Cornia
4. Giuseppe Fiameni
5. Matteo Stefanini
6. Silvia Cascianelli
7. Hanwang Zhang
8. Ron Mokady

In Natural Language Generation

1. Rita Cucchiara
2. Lorenzo Baraldi
3. Marcella Cornia
4. Giuseppe Fiameni
5. Matteo Stefanini
6. Ron Mokady
7. Amir Hertz
8. Silvia Cascianelli

Citations in the corpus

• ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Moreover, recent models apply self-attention [16, 43] or use an expressive visual Transformer [12] as an encoder [23].”
• More Grounded Image Captioning by Distilling Image-Text Matching Model (2020, background): “Recently, attention-based neural encoder-decoder models prevail [53, 56, 35, 6, 60, 29, 58, 59].”
• Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “Recent studies found that dataset bias, especially language bias, widely exists in vision-and-language tasks [14, 15, 24, 25, 26, 27, 28].”
• Visual Commonsense R-CNN (2020, unknown): “5 CNM [72] 37.”
• From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, methodology): “[75] used a self-attentive module to encode relationships between features coming from an object detector.”