Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering

By Soravit Changpinyo, Bo Pang, Piyush Sharma, Radu Soricut. 2019

20 citations recorded by the source corpus.

Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering. However, popular models like Faster R-CNN rely on a costly process of annotating ground-truths for both the bounding boxes and their corresponding semantic labels, making it less amenable as a primitive task for transfer learning. In this paper, we examine the effect of decoupling box proposal and featurization for down-stream tasks. The key insight is that this allows us to leverage a large amount of labeled annotations that were previously unavailable for standard object detection benchmarks. Empirically, we demonstrate that this leads to effective transfer learning and improved image captioning and visual question answering models, as measured on publicly-available benchmarks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Dacheng Tao
4. Trevor Darrell
5. Xiaogang Wang
6. Jianfeng Gao
7. Jitendra Malik
8. Andrew Zisserman

In Multimodal Learning

1. Lijuan Wang
2. Zhe Gan
3. Zicheng Liu
4. Jianfeng Wang
5. Xiaowei Hu
6. Cordelia Schmid
7. Arsha Nagrani
8. Jianfeng Gao

In Question Answering & Reading Comprehension

1. Radu Soricut
2. Piyush Sharma
3. Soravit Changpinyo
4. Paul Hongsuck Seo
5. Zhe Gan
6. Arsha Nagrani
7. Jianfeng Wang
8. Zicheng Liu

Citations in the corpus

• Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Thus, for our ic -based pre-training and downstream tasks, we implement a Transformer-based encoder-decoder model, using [19] as a starting point.”
• Connecting Vision and Language with Localized Narratives (2019, methodology): “We start from a state-of-the-art, transformer-based encoder-decoder image captioning model [3,9].”
• Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “Following [11], bounding boxes are proposed by a region proposal network (RPN) in [65] and featurized using Graph-Regularized Image Semantic Embeddings (Graph-RISE) [37].”
• i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2023, methodology): “Baselines include VisWiz Challenge Winner (Liu et al., 2021), BAN (Kim et al., 2018), B-Ultra & B-FRCNN (Changpinyo et al., 2019).”
• Injecting Semantic Concepts into End-to-End Image Captioning (2021, result): “We compare with the baseline methods FRCNN [8], Ultra [8] and [9].”
• Understanding Guided Image Captioning Performance across Domains (2020, methodology): “Following Changpinyo et al. (2019), we use Graph-Regularized Image Semantic Embed-ding (GraphRISE) as the global image features G.”