In Defense of Grid Features for Visual Question Answering
By Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Eric Learned-Miller, Xinlei Chen. 2020
368 citations recorded by the source corpus.
Popularized as `bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA). However, it is not clear whether the advantages of regions (e.g. better localization) are the key reasons for the success of bottom-up attention. In this paper, we revisit grid features for VQA, and find they can work surprisingly well -- running more than an order of magnitude faster with the same accuracy (e.g. if pre-trained in a similar fashion). Through extensive experiments, we verify that this observation holds true across different VQA models (reporting a state-of-the-art accuracy on VQA 2.0 test-std, 72.71), datasets, and generalizes well to other tasks like image captioning. As grid features make the model design and training process much simpler, this enables us to train them end-to-end and also use a more flexible network design. We learn VQA models end-to-end, from pixels directly to answers, and show that strong performance is achievable without using any region annotations in pre-training. We hope our findings help further improve the scientific understanding and the practical application of VQA. Code and features will be made available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Cordelia Schmid
- Ming-Hsuan Yang
- Jitendra Malik
- Yi Yang
In Multimodal Learning
In Question Answering & Reading Comprehension
Citations in the corpus
- FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “Another distinguishing factor between different models lies in the image features that are used, ranging from region features [63, 70, 118], to patch embeddings [54, 62, 109], to convolution or grid features [46, 51].”
- UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Over the past decade, we have witnessed a shift in visionlanguage research from task-specific models [13,15,39,44, 52] to large foundation models [1, 4, 21, 36, 45–47, 50].”
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, methodology): “Applying NMS to each and every class becomes a major runtime bottleneck with a large number of classes, e.g. 1.6K in the VG dataset (Jiang et al., 2020).”
- Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning (2021, methodology): “show that grid features can be equally effective as region features for VQA [17].”
- VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “Although [23] shows that the FPN model outperforms the C4 model for object detection, recent studies [14] demonstrate that FPN does not provide more effective region features for VL tasks than C4, which is also confirmed by our experimental results 2.”
- Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “Vision-language tasks, like VQA, image captioning, and image-text retrieval, also benefit from high-resolution feature maps [15, 49], which are extracted with pre-trained CNN models.”
- Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, background): “Increasing the resolution will likely further improve our performance as per the findings by [22].”
- MiniVLM: A Smaller and Faster Vision-Language Model (2020, methodology): “In [14], the grid feature extractor is constructed by casting the Faster-RCNN model into a fully-convolutional network and remove the region-related operations (e.”
- Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “There are other lines of VL pre-training works in which grid features [29, 32] are extracted from the convolutional layers without the proposal computation.”
- Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “Table 1 highlights a comparison between different aggregators, across two widely used image feature extractors in the literature [18] – Grid feature is the feature maps from ConvNets and Region feature is the ROI features from object detectors [1] (details in § 5).”
- X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020, methodology): “Our experiments in Sec 6 show that moving to a grid based input causes very little degradation to downstream QA tasks, a finding consistent with Jiang et al. (2020). Sampling grid features: Given text input, we sample predicted visual features {hvi}i=1 where T = N ×N is the number of image regions, ”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “appropriate for image captioning, ranging from the performance of better-trained grid features [88] to the emergence”
- KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, methodology): “The detector uses feature pyramid networks [45], and is trained using the hyper-parameters used for the baselines in [30].”
- Unified Graph Structured Models for Video Understanding (2021, background): “This approach is also known as “grid features” in visual question answering [4, 24].”
- Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts (2021, methodology): “The coarse-grained approach builds VLMs by extracting and encoding overall image features with convolutional network (Jiang et al., 2020; Huang et al., 2020; 2021) or vision”