Learning Answer Embeddings for Visual Question Answering

By Hexiang Hu, Wei-Lun Chao, Fei Sha. 2018

39 citations recorded by the source corpus.

We propose a novel probabilistic model for visual question answering (Visual QA). The key idea is to infer two sets of embeddings: one for the image and the question jointly and the other for the answers. The learning objective is to learn the best parameterization of those embeddings such that the correct answer has higher likelihood among all possible answers. In contrast to several existing approaches of treating Visual QA as multi-way classification, the proposed approach takes the semantic relationships (as characterized by the embeddings) among answers into consideration, instead of viewing them as independent ordinal numbers. Thus, the learned embedded function can be used to embed unseen answers (in the training dataset). These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers. We have also developed large-scale optimization techniques for applying the model to datasets with a large number of answers, where the challenge is to properly normalize the proposed probabilistic models. We validate our approach on several Visual QA datasets and investigate its utility for transferring models across datasets. The empirical results have shown that the approach performs well not only on in-domain learning but also on transfer learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “[32] propose a joint embedding where image-question representations can be matched with free-form answers.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “ht likely be closely embedded to the visual representation, given the question. To better utilize our learned multimodal representations specically for the VQA task, we use a similar architecture to [19], but for video. We learn two more sets of embeddings on top of the pre-trained embeddings that w”
Cross-Modal and Hierarchical Modeling of Video and Text (2018, methodology): “Examples are embedding images and the visual categories [8], embedding images and texts for VSE [19], and embedding images, questions, and answers for VQA [13].”
Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, background): “ImageQA has drawn significant research interests recently [2, 3, 8, 12, 30, 31, 39, 51, 52, 60].”
Synthesized Feature based Few-Shot Class-Incremental Learning on a Mixture of Subspaces (2021, methodology): “Note that instead of the traditional use of semantic word vectors in such a feature generation process [15, 29, 39, 43, 14], here we use sampled features to generate more features.”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “These methods can be categorized into three groups: joint learning mechanisms: BAN [26], Pythia [24], DFAF [15], fPMC [20], STL [54], CTI [11], and MCAN [59]; reasoning-based methods: Murel [4], ReGAT [30], MMN [6], NMS [22], and HAN [25]; and largescale visual-language modeling: LXMERT [47], OSCAR ”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “Typically, in [14], the authors learned two embedding functions to transform an image-question pair and an answer into a joint embedding space.”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, unknown): “cene graph. answer. In the past four years there are over a hundred of publications, but the improvement of the most advanced model over a simple multi-layer perceptron (MLP) baseline is merely 5 ˘7% [2,27,76]. Besides, the improvement is mostly built upon deep image and language features [24,25,52]”