Visual Question Generation as Dual Task of Visual Question Answering

By Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang. 2017

177 citations recorded by the source corpus.

Visual question answering (VQA) and visual question generation (VQG) are two trending topics in the computer vision, but they are usually explored separately despite their intrinsic complementary relationship. In this paper, we propose an end-to-end unified model, the Invertible Question Answering Network (iQAN), to introduce question generation as a dual task of question answering to improve the VQA performance. With our proposed invertible bilinear fusion module and parameter sharing scheme, our iQAN can accomplish VQA and its dual task VQG simultaneously. By jointly trained on two tasks with our proposed dual regularizes (termed as Dual Training), our model has a better understanding of the interactions among images, questions and answers. After training, iQAN can take either question or answer as input, and output the counterpart. Evaluated on the CLEVR and VQA2 datasets, our iQAN improves the top-1 accuracy of the prior art MUTAN VQA method by 1.33% and 0.88% (absolute increase) respectiely. We also show that our proposed dual training framework can consistently improve model performances of many popular VQA architectures.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “Visual Question Generation (Shukla et al., 2019; Li et al., 2018) 28.”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, background): “Recently, we have seen an increasing interest in the QG area, with mainly three categories: Text-based QG (Du et al., 2017; Zhao et al., 2018), Knowledge-Base-based QG (Reddy et al., 2017; Serban et al., 2016), and Image-based QG (Li et al., 2018; Jain et al., 2017).”
Towards a Better Metric for Evaluating Question Generation Systems (2018, background): “…problem and create even more training data, there is growing interest in developing techniques that can automatically generate questions from a given source, say a document (Du et al., 2017; Du and Cardie, 2017), knowledge base (Reddy et al., 2017; Serban et al., 2016), or image (Li et al., 2017) .”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “The methods in [52] and [69] propose to jointly learn VQG and VQA to improve the image VQA task.”
Hybrid Reinforcement Learning with Expert State Sequences (2019, background): “However, as adopted by Li et al. (2017), the proposed estimation gives better empirical results.”
What BERT Sees: Cross-Modal Transfer for Visual Question Generation (2020, methodology): “…contains 2500 training images, 1250 validation images and 1250 test images from MS COCO (Lin et al., 2014); each image has 5 corresponding questions and 5 ground-truth captions.3 The Visual Question Answering (V QA) (Antol et al., 2015) dataset can be used to derive VQG data (Li et al., 2018).”
Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering (2018, methodology): “[20], introduce VQA and VQG as dual tasks and suggest a joint training for the two tasks.”
All You May Need for VQA are Image Captions (2022, background): “…2018; Uehara et al., 2018; Krishna et al., 2019), e.g., for them to to be answered by humans, or to help improving the VQA task (Kaﬂe et al., 2017; Li et al., 2018; Shah et al., 2019; Xu et al., 2021; Kil et al., 2021; Akula et al., 2021), e.g., for additional evaluation and as means of data…”
Factor Graph Attention (2019, methodology): “Image captioning [36, 56, 24, 50, 7, 4, 13] was an instrumental language+vision task, followed by visual question answering [33, 42, 25, 34, 41, 5, 15, 59, 23, 3, 9, 14, 46, 55, 42, 54, 39, 40, 43] and visual question generation [41, 38, 22, 49, 28, 6].”
Small Sample Learning in Big Data Era (2018, methodology): “For model agnostic training procedure, Li et al. (2018a) made metalearning on simulated train/test split with domain-shift for domain generalisation, which could be applied to different base network types.”
Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards (2017, background): “As a result of the above, Visual Question Generation (VQG) has started to receive research attention, but primarily as a vision-to-language problem [16, 20, 32].”
Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering (2017, background): “Great progress has been achieved for different tasks including image captioning (Karpathy and Fei-Fei 2015), visual question generation (Mostafazadeh et al. 2016; Li et al. 2017b), video question answering (Ye et al. 2017) and text-to-image retrieval (Xie, Shen, and Zhu 2016; Li et al. 2017a).”
Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks (2021, background): “…widely applied in various ﬁelds, such as machine translation (He et al., 2016), sentiment classiﬁcation (Xia et al., 2017), question answering (Tang et al., 2017), visual question answering (Li et al., 2018), machine reading comprehension (Xiao et al., 2018), and code generation (Wei et al., 2019).”
Code Generation as a Dual Task of Code Summarization (2019, background): “Li et al. [2018] applied a dual framework to improve the performance of visual question answering with the help of visual question generation.”
Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering (2019, background): “[30] proposed the invertible Question Answering Network (iQAN) to leverage the complementary relations between questions and answers in the image by jointly learning the Visual Question Answering (VQA) and Visual Question Generation (VQG) tasks.”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, background): “red. Therefore, it is non-trivial to explore the data augmentation technique to facilitate VQA. Previous works have generated reasonable questions based on the image content [16] and the given answer [25], namely Visual Question Generation (VQG). However, a signicant portion of the generated questio”
Question-Guided Hybrid Convolution for Visual Question Answering (2018, methodology): “Recently, CNN and RNN are jointly trained for learning feature representations for multi-modal tasks, including image captioning [3, 4], text-to-image retrieval [5, 34], and Visual Question Answering (VQA) [6, 26, 40].”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “V&L Multi-Task Learning Instead of building a monolithic model for different V&L tasks, numbers of previous work explore multi-task learning with a unified model for utilising the common and the complementary knowledge to reduce the domain gap [31, 39, 42, 47, 33].”