Learning to Count Objects in Natural Images for Visual Question Answering

By Yan Zhang, Jonathon S. Hare, Adam Prügel-Bennett. 2018

224 citations recorded by the source corpus.

Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Towards VQA Models That Can Read (2019, neutral): “Counter [50] 68.”
Bilinear Attention Networks (2018, methodology): “BAN-Glove-Counter uses both the previous 600-dimensional word embeddings and counting module [41], which exploits spatial information of detected object boxes from the feature extractor [2].”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, methodology): “number type) to BAN+Counter, and in doing so does not use any auxiliary information like the bounding-box coordinates of each object [35].”
In Defense of Grid Features for Visual Question Answering (2020, background): “, counting module [52, 41] in the bottom right example) to answer the question correctly.”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “Compared to the popular bi-linear fusion [58], Eq (10) is empirically shown a faster convergence of training the label embeddings in our experiments.”
Errudite: Scalable, Reproducible, and Testable Error Analysis (2019, methodology): “VQACounting is built on top of SAAA, with increased performance on counting questions.”
Are Red Roses Red? Evaluating Consistency of Question-Answering Models (2019, methodology): “Figure 1: Inconsistent QA Predictions: Models that are accurate for questions from these datasets (ﬁrst row in (b) and (d)) are not able to correctly answer follow-up questions whose answers are implied by the original question/answer.”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, background): “[31] in the context of counting, but we note a more general problem.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “Other kind of approaches such as HDGAN (Zhang et al., 2018b) aim to accompany the hierarchical adversarial objectives inside the network to regularize mid-level representations and assist generator training to capture the complex image information.”
Multi-Modality Latent Interaction Network for Visual Question Answering (2019, methodology): “Counter [54] dives into the number question of VQA by utilising the relative position between bounding box for learning efﬁcient Non Maximum Suppression(NMS).”
Feature Pyramid Transformer (2020, background): “Moreover, it has been empirically shown that the negative value of the euclidean distance Feud is more effective in computing the similarity than dot product when the semantic information of two feature maps is different [35].”
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (2019, background): “Incorporating object detection signals in down-stream tasks appropriately is non-trivial and an active subject for research (Santoro et al., 2017; Zhang et al., 2018).”
Obtaining Faithful Interpretations from Compositional Neural Networks (2020, methodology): “Graph-count module (Zhang et al., 2018) is a middle ground between both approaches - the naı̈ve Sum-Count and the flexible Layer-Count.”
GPS-Net: Graph Property Sensing Network for Scene Graph Generation (2020, background): “∗ represents a fusion function defined in [47]: x ∗ y = ReLU (Wxx + Wyy)−(Wxx − Wyy)⊙(Wxx − Wyy), where Wx and Wy project x, y to 1024-dimensional space, respectively.”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, background): “A large portion of prior arts [19,46,29,31] are built upon the bottom-up top-down (BUTD) attention method [2].”
Bilinear Graph Networks for Visual Question Answering (2019, methodology): “With the extra Counter module [53] in our BGNs+Glove+Counter model, we make another 3% gain on number metric, which means that our method and Counter”
Learning Conditioned Graph Structures for Interpretable Visual Question Answering (2018, methodology): “We compare our model to [19] (ReasonNet), the approach by [3] (Bottom-Up) who won the 2017 VQA challenge 1, and the recent approach proposed in [11] (Counting module) which focuses on optimising counting questions.”