Dual-Level Collaborative Transformer for Image Captioning
By Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, Rongrong Ji. 2021
360 citations recorded by the source corpus.
Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features. Concretely, in DLCT, these two features are first processed by a novel Dual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr on Karpathy split and 135.4% CIDEr on the official split.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Giuseppe Fiameni
- Rita Cucchiara
- Marcella Cornia
- Lorenzo Baraldi
- Silvia Cascianelli
- Matteo Stefanini
- Lior Wolf
- Yoav Shalev
In Computer Vision and Pattern Recognition
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Rongrong Ji
- Silvia Cascianelli
- Feiyue Huang
In Multimodal Learning
- Giuseppe Fiameni
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Silvia Cascianelli
- Rongrong Ji
- Xiaoshuai Sun
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Lior Wolf
- Ron Mokady
- Silvia Cascianelli
Citations in the corpus
- ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Early works have used LSTM variants [8, 38, 39], while recent works [16, 26] adopted the improved transformer architecture [36].”
- ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “On the text modeling side of the problem, language models (LMs) have also advanced with the development of LSTMs [16, 69], CNNs [4] and Transformers [27, 29, 47].”
- From Show to Tell: A Survey on Image Captioning (2021, methodology): “Luo et al. [73] proposed a hybrid approach that combines region and grid features to exploit their complementary advantages.”
- RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, methodology): “The main development of image captioning [41, 40, 5, 14, 24] can be divided into two stages: traditional method stage and deep learning method stage.”
- Multiple Transformer Mining for VizWiz Image Caption (2021, methodology): “Now there are many methods based on encoding and decoding in image captioning[8][11][6][3][5].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “tural modifications [77], [78], [86], [94].”
- Multimodal Learning With Transformers: A Survey (2022, background): “graph [228], video to caption [229], [230], [231], [232], image to caption [233], [234], [235], [236], [237], text to speech [238], text to image [205], [239], text to shape [240], RGB to 3D human”