In Defense of Grid Features for Visual Question Answering

By Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Eric Learned-Miller, Xinlei Chen. 2020

368 citations recorded by the source corpus.

Popularized as `bottom-up' attention, bounding box (or region) based visual features have recently surpassed vanilla grid-based convolutional features as the de facto standard for vision and language tasks like visual question answering (VQA). However, it is not clear whether the advantages of regions (e.g. better localization) are the key reasons for the success of bottom-up attention. In this paper, we revisit grid features for VQA, and find they can work surprisingly well -- running more than an order of magnitude faster with the same accuracy (e.g. if pre-trained in a similar fashion). Through extensive experiments, we verify that this observation holds true across different VQA models (reporting a state-of-the-art accuracy on VQA 2.0 test-std, 72.71), datasets, and generalizes well to other tasks like image captioning. As grid features make the model design and training process much simpler, this enables us to train them end-to-end and also use a more flexible network design. We learn VQA models end-to-end, from pixels directly to answers, and show that strong performance is achievable without using any region annotations in pre-training. We hope our findings help further improve the scientific understanding and the practical application of VQA. Code and features will be made available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Zhe Gan
  3. Mohit Bansal
  4. Lijuan Wang
  5. Dacheng Tao
  6. Zicheng Liu
  7. Yejin Choi
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Trevor Darrell
  5. Cordelia Schmid
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Yi Yang

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Xiaowei Hu
  6. Jianfeng Gao
  7. Zhengyuan Yang
  8. Linjie Li

In Question Answering & Reading Comprehension

  1. Weizhe Lin
  2. Bill Byrne
  3. Lijuan Wang
  4. Jianfeng Gao
  5. Xiaowei Hu
  6. Jianwei Yang
  7. Zhengyuan Yang
  8. Zicheng Liu

Citations in the corpus