Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering

By Ruixue Tang, Chao Ma, W. Zhang, Qi Wu, Xiaokang Yang. 2020

56 citations recorded by the source corpus.

Visual Question Answering (VQA) has achieved great success thanks to the fast development of deep neural networks (DNN). On the other hand, the data augmentation, as one of the major tricks for DNN, has been widely used in many computer vision tasks. However, there are few works studying the data augmentation problem for VQA and none of the existing image based augmentation schemes (such as rotation and flipping) can be directly applied to VQA due to its semantic structure -- an $\langle image, question, answer\rangle$ triplet needs to be maintained correctly. For example, a direction related Question-Answer (QA) pair may not be true if the associated image is rotated or flipped. In this paper, instead of directly manipulating images and questions, we use generated adversarial examples for both images and questions as the augmented data. The augmented examples do not change the visual properties presented in the image as well as the \textbf{semantic} meaning of the question, the correctness of the $\langle image, question, answer\rangle$ is thus still maintained. We then use adversarial learning to train a classic VQA model (BUTD) with our augmented data. We find that we not only improve the overall performance on VQAv2, but also can withstand adversarial attack effectively, compared to the baseline model. The source code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jingjing Liu
3. Linjie Li
4. Mohit Bansal
5. Jianfeng Gao
6. Eduard H. Hovy
7. Varun Prashant Gangal
8. Soroush Vosoughi

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Dacheng Tao
5. Luc Van Gool
6. Jianfeng Gao
7. Mohit Bansal
8. Cordelia Schmid

In Natural Language Processing

1. Zhe Gan
2. Jingjing Liu
3. Varun Prashant Gangal
4. Eduard H. Hovy
5. Teruko Mitamura
6. Soroush Vosoughi
7. Linjie Li
8. Jason Wei

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Jason Wei
5. Sarath Chandar
6. Lijuan Wang
7. Jianfeng Gao
8. Yu Cheng

Citations in the corpus

• A Survey of Data Augmentation Approaches for NLP (2021, background): “Kafle et al. (2017), Yokota and Nakayama (2018), and Tang et al. (2020) pro-pose methods for visual QA including question generation and adversarial examples.”
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “Recently, adversarial training (AT) [69, 74, 55, 50, 79, 71] has shown success on standard V+L tasks [16, 65].”