Data Augmentation for Visual Question Answering

By Kushal Kafle, Mohammed A. Yousefhussien, Christopher Kanan. 2017

116 citations recorded by the source corpus.

Data augmentation is widely used to train deep neural networks for image classification tasks. Simply flipping images can help learning tremendously by increasing the number of training images by a factor of two. However, little work has been done studying data augmentation in natural language processing. Here, we describe two methods for data augmentation for Visual Question Answering (VQA). The first uses existing semantic annotations to generate new questions. The second method is a generative approach using recurrent neural networks. Experiments show that the proposed data augmentation improves performance of both baseline and state-of-the-art VQA algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Automatically Learning Data Augmentation Policies for Dialogue Tasks (2019, neutral): “…Lemley et al., 2017) and NLP (F¨urstenau and Lapata, 2009; Sen-nrich et al., 2016; Wang and Yang, 2015; Zhang et al., 2015; Jia and Liang, 2016; Kim and Rush, 2016; Hu et al., 2017; Xu et al., 2017; Xia et al., 2017; Silfverberg et al., 2017; Kaﬂe et al., 2017; Hou et al., 2018; Wang et al., 2018).”
An Analysis of Visual Question Answering Algorithms (2017, methodology): “Sport recognition, counting, subordinate object recognition, object presence, scene understanding, positional reasoning, and absurd questions were created from COCO, similar to the scheme used in [15].”
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks (2019, methodology): “We suspect that EDA will work with any thesaurus.”
Cycle-Consistency for Robust Visual Question Answering (2019, background): “Previous attempts [19] at naively generating questions conditioned on the answer and using them without ﬁltering to augment the training data have been unsuccessful.”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, methodology): “In traditional data augmentation settings, augmented data are usually mixed with the original training set and are trained together with the original data (Wei and Zou, 2019; Kafle et al., 2017).”
VQD: Visual Query Detection In Natural Scenes (2019, methodology): “The questions are generated using multiple templates for each question type, which is an approach that has been used in earlier work for VQA (Kafle and Kanan, 2017a; Kafle et al., 2017).”
SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (2020, unknown): “…(2015); Wei and Zou (2019) utilize heuristic approaches including synonym replancement, random insertion, swap and deletion for text augmentation, Kaﬂe et al. (2017); Silfverberg et al. (2017) employ heuristic rules based on speciﬁc task, Hu et al. (2017) pro-pose to augment text data in an…”
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020, methodology): “Existing task-specific methods (Kafle et al., 2017) and word-level methods (Zhang et al., 2015; Xie et al., 2017; Wei and Zou, 2019) are based on human-designed heuristics.”
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (2022, methodology): “, for visual QA (Kafle et al., 2017) and textual QA (Lewis et al.”
Data augmentation techniques for the Video Question Answering task (2020, methodology): “made of either an object, or an object and an attribute, or two objects and a relation between them, from which new question-answer pairs are constructed using previously built templates. Ka e et al. [14] proposes two techniques. The rst is a template-based method which exploits the COCO dataset [21”
DVQA: Understanding Data Visualizations via Question Answering (2018, methodology): “nsfer from synthetic to real-world bar charts. Training on entirely real charts would be ideal, but even then there would likely be a beneﬁt to using synthetic datasets as a form of data augmentation [17]. 7. Conclusion Here, we described DVQA, a dataset for understanding bar charts. We demonstrated”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, background): “ver, our study shows that it could degrade the model performance on the VQA task (see Section4). Other works generate paraphrases [45,28] and add noise to smooth text data [42]. There are fewer works [18,33,35,1,30] that learn data augmentation for VQA. Ka e et al.[18] do a pioneer work where they g”
A Survey on Data Augmentation for Text Classification (2021, unknown): “Research in this area was therefore much more limited before 2019, despite existing extensive areas of application [3].”
Substructure Substitution: Structured Data Augmentation for NLP (2021, background): “3496 Another approach to example generation is to generate new examples based on predefined templates (Kafle et al., 2017; Asai and Hajishirzi, 2020), where the templates are designed following heuristic, and usually task-specific, rules.”