CPTR: Full Transformer Network for Image Captioning
By Wei David Liu, Sihan Chen, Longteng Guo, Xinxin Zhu. 2021
172 citations recorded by the source corpus.
In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the"CNN+Transformer"design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional"CNN+Transformer"methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the"words-to-patches"attention in the decoder thanks to the full Transformer architecture.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaodan Liang
- Xiaogang Wang
- Yi Yang
- Christoph Feichtenhofer
In Multimodal Learning
- Giuseppe Fiameni
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Matteo Stefanini
- Silvia Cascianelli
- W. Zhang
- Jitendra Malik
In Model Architectures
- Christoph Feichtenhofer
- Yanghao Li
- Haoqi Fan
- Bo Xiong
- Jitendra Malik
- Karttikeya Mangalam
- Lewei Yao
- W. Zhang
Citations in the corpus
- ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “More-over, recent models apply self-attention [16, 43] or use an expressive visual Transformer [12] as an encoder [23].”
- Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [115, 99], 3D reconstruction [72], pose estimation [107], generative modeling [14], image retrieval [27], medical image segmentation [13, 97, 111], point clouds [40], video in”
- Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (2021, background): “object re-identification [41], image captioning [42], point cloud learning [43] and self-supervised learning [44].”
- From Show to Tell: A Survey on Image Captioning (2021, methodology): “Nonetheless, the surveyed work [77] explores a fully-Transformer paradigm, where image patches are directly applied to Transformer encoders, as in the Vision Transformer proposed in [75].”
- Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, background): “The recent advances and trends of MMDL are from Audio-visual speech recognition (AVSR) [173], multimodal emotion recognition [26], image and video captioning [58, 89], Visual Question-Answering (VQA) [161], to multimedia retrieval [134] and so on.”
- DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment (2023, background): “, VQA [2, 15, 24, 28] and image captioning [1, 30, 45, 50], etc.”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “[92] devised the first convolution-free architecture for image”