Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
By Damien Teney, Peter Anderson, Xiaodong He, Anton van den Hengel. 2017
402 citations recorded by the source corpus.
Deep Learning has had a transformative impact on Computer Vision, but for all of the success there is also a significant cost. This is that the models and procedures used are so complex and intertwined that it is often impossible to distinguish the impact of the individual design and engineering choices each model embodies. This ambiguity diverts progress in the field, and leads to a situation where developing a state-of-the-art model is as much an art as a science. As a step towards addressing this problem we present a massive exploration of the effects of the myriad architectural and hyperparameter choices that must be made in generating a state-of-the-art model. The model is of particular interest because it won the 2017 Visual Question Answering Challenge. We provide a detailed analysis of the impact of each choice on model performance, in the hope that it will inform others in developing models, but also that it might set a precedent that will accelerate scientific progress in the field.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Zhe Gan
- Dacheng Tao
- Christopher D. Manning
- Devi Parikh
- Percy Liang
- Yejin Choi
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Yi Yang
- Cordelia Schmid
- Andrew Zisserman
In Multimodal Learning
In Question Answering & Reading Comprehension
- Christopher Kanan
- Kushal Kafle
- Anton van den Hengel
- Damien Teney
- Dacheng Tao
- Zhou Yu
- Christopher D. Manning
- Drew A. Hudson
Citations in the corpus
- Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021, methodology): “To fine-tune for VQA, we replace the image–text matching loss with a 2-layer MLP and train with a binary cross-entropy loss against soft answer scores (Teney et al., 2018).”
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “– [49] presents various other tips and tricks which work well for the task of VQA, for instance: sigmoid outputs (instead of softmax), soft training targets, gated tanh activations, answer embeddings initialized using GloVe and Google Images, large mini-batches, and smart shuffling of training data.”
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, background): “[41] for further details of the VQA model implementation.”
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [16, 29, 30, 15].”
- Bilinear Attention Networks (2018, methodology): “Model Overall Yes/no Number Other Test-std Bottom-Up [2, 29] 65.”
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “0 [23], CLEVR [32], Visual7w [81], Visual Genome [41], COCO-QA [57], and models [33, 60, 29, 31, 1, 67, 17, 37] have been used to measure the progress in understanding the interaction between vision and language modalities.”
- Deep Modular Co-Attention Networks for Visual Question Answering (2019, methodology): “In contrast to [28] which only uses the final state (i.”
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018 (2018, methodology): “ grid level features helps to further improve the performance to 69.81%. Instead of using an adaptive protocol for choosing the number of object proposals (between 10 and 100) per image as as done in [14], we also experimented with using a simpler (but slower) strategy of using 100 objects proposals”
- Stacking with Auxiliary Features for Visual Question Answering (2018, background): “Visual Question Answering (VQA), the task of addressing open-ended questions about images (Ma-linowski and Fritz, 2014; Antol et al., 2015), has attracted significant attention in recent years (Andreas et al., 2016a; Goyal et al., 2016; Agrawal et al., 2016; Teney et al., 2017).”
- GQA: a new dataset for compositional question answering over real-world images (2019, background): “Other works have argued for the need to devise new evaluation measures and techniques to shed more light on systems’ inner workings [18, 35, 36, 17].”
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “Compared to the image features extracted from VGG-Net [42] and ResNet [19], this type of features has achieved state-of-the-art performance in both image captioning and VQA [1, 49] tasks.”
- Learning to Count Objects in Natural Images for Visual Question Answering (2018, methodology): “Additionally, we can evaluate the accuracy over balanced pairs as proposed by Teney et al. (2017): the ratio of balanced pairs on which the VQA accuracy for both questions is 1.0.”
- Did the Model Understand the Question? (2018, background): “The Visual Question Answering Task (Agrawal et al., 2015; Teney et al., 2017; Kazemi and Elqursh, 2017; Ben-younes et al., 2017; Zhu et al., 2016) requires a system to answer questions about images (fig.”
- Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “…with high magnitudes correspond to image regions which contain important semantic content. to handle many objects and their complex relations while also integrating rich background knowledge, and attention has emerged as a promising strategy for achieving good performance [7,8,9,10,11,12,13,14].”
- Visual Entailment Task for Visually-Grounded Language Learning (2018, methodology): “Lastly, the split-transform-merge technique follows from the VQA 2017 winner [Teney et al., 2017].”
- Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “ beyond the question biases that VQA-CP was specically designed for. Results We present results of our method applied on top of two existing models. The rst (weak baseline) is the popular BUTD model [3,64]. The second (strong baseline) is the \unshuing" method of [17], which was specically tune”
- Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, methodology): “t proposals from Faster R-CNN [11], and the top-down mechanism predicting an attention distribution over those proposals. This model obtained rst place in the 2017 VQA Challenge. Following setting in [2,39], we use a maximum of 100 object proposals per image, which are 2048 dimensional features, as ”