Visual Question Answering: From Theory to Application

By Allan Jabri, Armand Joulin, L. V. D. van der Maaten. 2022

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “2015), biases like “2” being the correct answer to 39% of the questions starting with “how many” allowed learning algorithms to perform well while ignoring the visual modality altogether (Jabri et al., 2016; Goyal et al., 2017).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “…analyses showing this type of effects include visual question answering (VQA) systems performing well even when the image is not available (Jabri et al., 2016), a benchmark for theory of mind in conversational AI systems being solvable without extracting any information about agents…”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “Similar problems were found in VQA [3], where simple baselines without sophisticated multimodal understanding performed remarkably well [94, 35, 1, 26] and in multimodal machine translation [18, 74], where the image was found to matter relatively little [13, 17, 7].”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “• Unsurprisingly, LSTM-only , CNN-only and also CNN+BoW:Mult are not able to learn actual multimodal understanding, in contrast to their good performance on real-world data (Jabri et al., 2016).”
Compact Trilinear Interaction for Visual Question Answering (2019, methodology): “There are other works that consider answer information, besides image and question information, to improve VQA performance [16, 9, 36, 14, 41, 34].”
Learning from others' mistakes: Avoiding dataset biases without modeling them (2020, background): “Examples include visual question answering (Jabri et al., 2016; Zhang et al., 2016), story completion (Schwartz et al., 2017), and reading comprehension (Kaushik & Lipton, 2018; Chen et al., 2016).”
Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, background): “But such evaluation is not straightforward; for example, an early Visual Question-Answering (VQA) challenge was later “broken” by a high-performing method that ignored the image entirely (Jabri et al., 2016).”
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation (2021, background): “, 2017), data bias (Jabri et al., 2016), and some others (Dodge et al.”
Data Efficient Masked Language Modeling for Vision and Language (2021, unknown): “…on VQA datasets using strong language priors, and by relying on superficial cues, and there are still challenges to overcome for tasks with more compositional structure (Jabri et al., 2016; Zhang et al., 2016; Goyal et al., 2017; Agarwal et al., 2020; Bitton et al., 2021; Dancette et al., 2021).”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “However, designing such a system can contain several other challenges such as coming up with strong baselines (Jabri et al., 2016).”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, background): “Despite VQA is a multimodal task, a large body of research [21, 2, 42, 17] has shown the existence of language biases in VQA.”
KB-VLP: Knowledge Based Vision and Language Pretraining (2021, background): “Following Antol et al. (2015), the model is required to answer natural language questions based on an image.”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, background): “…to decompose a question into subquestions (Min et al., 2019b; Perez et al., 2020), to access the image for image-based question-answering (Antol et al., 2015; Zhang et al., 2016a), and to view the premise when detecting if it entails a hypothesis (Gururangan et al., 2018; Poliak et al.,…”
FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding (2020, methodology): “Most of the recent VQA algorithms have trained on natural image based datasets such as DAQUAR[62], COCO-VQA [4], Visual Genome[40], Visual7W [79].”
Imagination Improves Multimodal Translation (2017, background): “It has previously been shown that the choice of visual features can affect the performance of vision and language models (Jabri et al., 2016; Kiela et al., 2016).”