Learning Visual Reasoning Without Strong Priors

By Ethan Perez, Harm de Vries, Florian Strub, Vincent Dumoulin, Aaron C. Courville. 2017

65 citations recorded by the source corpus.

Achieving artificial visual reasoning - the ability to answer image-related questions which require a multi-step, high-level process - is an important step towards artificial general intelligence. This multi-modal task requires learning a question-dependent, structured reasoning process over images from language. Standard deep learning approaches tend to exploit biases in the data rather than learn this underlying structure, while leading methods learn to visually reason successfully but are hand-crafted for reasoning. We show that a general-purpose, Conditional Batch Normalization approach achieves state-of-the-art results on the CLEVR Visual Reasoning benchmark with a 2.4% error rate. We outperform the next best end-to-end method (4.5%) and even methods that use extra supervision (3.1%). We probe our model to shed light on how it reasons, showing it has learned a question-dependent, multi-step process. Previous work has operated under the assumption that visual reasoning calls for a specialized architecture, but we show that a general architecture with proper conditioning can learn to visually reason effectively.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Conditional normalization layers include the Conditional Batch Normalization (Conditional BN) [12] and Adaptive Instance Normalization (AdaIN) [19], Both were first used in the style transfer task and later adopted in various vision tasks [5,9,11,20,25,34,36,39,46,51].”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, methodology): “In this paper, which expands upon a shorter report (Perez et al. 2017), our key contribution is that we show FiLM is a strong conditioning method by showing the following on visual reasoning tasks:”
Semantics Disentangling for Text-To-Image Generation (2019, methodology): “These parameters are used to control the behavior of the main network for the tasks such as image stylization [15], visual reasoning [27], video segmentation [38], question answering [5] and etc.”
Slimmable Neural Networks (2018, methodology): “Conditional normalization is widely used in tasks including style transfer (Dumoulin et al., 2016; Li et al., 2017a; Huang & Belongie, 2017; Li et al., 2017b), image recognition (Li et al., 2016b; Yang et al., 2018) and many others (Perez et al., 2017b;a).”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, methodology): “Given the large number of highperforming models on CLEVR [14, 24, 5, 19], we train our model 5 times for a statistical measure of performance, achieving mean validation accuracy of 99.”
Overcoming Language Priors in Visual Question Answering with Adversarial Regularization (2018, background): “In response to these challenges, there has been extensive work on VQA in recent years both in terms of dataset curation [6, 12, 2, 17, 13, 32, 3] and modeling [2, 5, 28, 14, 16, 4, 22, 20].”
Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform (2018, background): “Some variants of CN have proven highly effective in image style transfer [12, 17, 15], visual question answering [6] and visual reasoning [35].”
Object Level Visual Reasoning in Videos (2018, background): “From a diﬀerent perspective, [27] show that relational reasoning can be learned for visual reasoning in a data driven way without any prior using conditional batch normalization with a feature-wise aﬃne transformation based on conditioning information.”
DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer (2018, background): “Both our result and FiLM were obtained on single graphics cards and were only limited in number of epochs for practicality.”
FigureQA: An Annotated Figure Dataset for Visual Reasoning (2017, methodology): “To measure the smoothness of curves for question templates 9 and 10, we devised a roughness metric based on the sum of absolute pairwise differences of slopes, computed via ﬁnite differences.”
Neural Compositional Denotational Semantics for Question Answering (2018, background): “executing the complete sentence encoding against a knowledge source (Perez et al., 2017).”
Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation (2019, methodology): “Similarly, employing FiLM modulation has been shown effective on a variety of tasks such as image synthesis [19, 36, 17, 1], visual question answering [38, 37], style transfer [7], recognition [16, 60], reading comprehension [5], etc.”
Differentiable Learning-to-Normalize via Switchable Normalization (2018, background): “We would also like to acknowledge the contributions of previous work that explored spatial region (Ren et al., 2016) and conditional normalization (Perez et al., 2017).”