Creativity: Generating Diverse Questions Using Variational Autoencoders

By Unnat Jain, Ziyu Zhang, Alexander Gerhard Schwing. 2017

159 citations recorded by the source corpus.

Generating diverse questions for given images is an important task for computational education, entertainment and AI assistants. Different from many conventional prediction techniques is the need for algorithms to generate a diverse set of plausible questions, which we refer to as creativity. In this paper we propose a creative algorithm for visual question generation which combines the advantages of variational autoencoders with long short-term memory networks. We demonstrate that our framework is able to generate a large set of varying questions given a single input image.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, background): “Recently, we have seen an increasing interest in the QG area, with mainly three categories: Text-based QG (Du et al., 2017; Zhao et al., 2018), Knowledge-Base-based QG (Reddy et al., 2017; Serban et al., 2016), and Image-based QG (Li et al., 2018; Jain et al., 2017).”
Towards a Better Metric for Evaluating Question Generation Systems (2018, background): “Apart from generating questions from text there is also research on generating questions from images (Jain et al., 2017; Li et al., 2017; Zhang et al., 2017) and knowledge base (Serban et al., 2016; Reddy et al., 2017).”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering (2019, background): “Neural alternatives to visual question generation have been recently proposed [25, 12, 35], but they aim at a quite different goal of creating engaging but potentially inaccurate questions about the wider context of the image such as subjective evoked feelings or speculative events that may lead to ”
GQA: a new dataset for compositional question answering over real-world images (2019, background): “2 Neural alternatives to visual question generation have been recently proposed [29, 14, 42], but they aim at a quite different goal of creating “interesting and engaging” questions about the wider context of the image, e.”
Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering (2020, background): “Previous works have generated reasonable questions based on the image content [16] and the given answer [25], namely Visual Question Generation (VQG).”
FLIPDIAL: A Generative Model for Two-Way Visual Dialogue (2018, background): “LSTM [10], GRU [5]) in modelling both visual dialogue and general dialogue to date [6, 7, 8, 12, 27].”
Two Can Play This Game: Visual Dialog with Discriminative Question Generation and Answering (2018, result): “ creating a `natural and engaging' question. Recently, Vijayakumar et al. [37] have shown preliminary results for this task as well. In contrast to the two aforementioned techniques, Jain et al. [15] argued for more diverse predictions and employed a variational auto-encoder approach. Work by L”
Question Asking as Program Generation (2017, background): “For instance, an image of a car wreck might be paired with the question, “What caused the accident?” Deep neural networks, similar to those used for image captioning, are capable of producing these types of questions after extensive training [16, 23, 11].”
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning (2019, background): “Image captioning and paragraph generation [3, 4, 6, 7, 12–14, 19–22, 24, 25, 30, 36, 39, 41] have attracted a significant amount of work.”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, background): “Other algorithms learn to condition generation on latent variables during training (Bowman et al., 2016), which are sampled from at inference time to encourage diversity (Jain et al., 2017; Gupta et al., 2018; Park et al., 2019).”
What BERT Sees: Cross-Modal Transfer for Visual Question Generation (2020, background): “Zhang et al. (2017) aim at generating, for a given image, multiple visually grounded questions of varying types (what, when, where, etc.); similarly, Jain et al. (2017) generate diverse questions using Variational Autoencoders.”
Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering (2018, background): “Using a variety of datasets [11, 2, 8, 10, 3, 5], models based on multi-modal representation and attention [18–25], deep network architectures [26, 12, 27, 28], and dynamic memory nets [29] have been developed.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “or powerful deep generative models, such as GANs and VAEs [47], [237]–[239].”
Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images (2017, unknown): “We note that generating diverse sentences is also a challenge in visual question generation, see concurrent work [19], and in language-only dialogue generation studied in the linguistic community, see e.”
De-Confounded Variational Encoder-Decoder for Logical Table-to-Text Generation (2021, methodology): “In the vanilla CVAE formulation, such as the ones adopted in (Kingma et al., 2014; Jain et al., 2017), the prior distribution p(z|x) is approximated to p(z), which is independent on x and fixed to a zero-mean unit-variance Gaussian distribution N (0, I).”
Convolutional Image Captioning (2017, background): “LSTM nets have been considered as the de-facto standard for vision-language tasks of image captioning [5, 16, 37, 39, 38], visual question answering [3, 30, 28], question generation [14, 20], and visual dialog [7, 13], due to their compelling ability to memorize long-term depen-”