Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

By Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Josh Tenenbaum. 2018

717 citations recorded by the source corpus.

We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Grammar-Based Grounded Lexicon Learning (2022, methodology): “(3) Similarly, NS-VQA [46] also parses the question into a symbolic program.”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “Another attempt to bridge the gap comes from the neurosymbolic perspective (Gupta et al., 2019; Hudson and Manning, 2019; Mao et al., 2019; Yi et al., 2018).”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, methodology): “Each model consists of a bidirectional LSTM encoder plus an LSTM decoder [48].”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “Posterior efforts have focused on explicitly reasoning about the composition by separating the reasoning from the perception, with connections to neuro-symbolic meth-ods [19, 27, 62].”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, methodology): “Several methods have been proposed for compositional visual reasoning, including modular neural networks (e.g., Andreas et al., 2016; Johnson et al., 2017b; Perez et al., 2018; Hu et al., 2017; Suarez et al., 2018; Hu et al., 2018; Yao et al., 2018; Yi et al., 2018) and attention- or memory-based me”
In Defense of Grid Features for Visual Question Answering (2020, background): “Today, most state-of-the-art VQA models focus on fusing schemes [49, 20, 48] and are built with region features asis [47]; whereas our work revisits grid features, and shows that they can be equally effective and lead to remarkable speed-ups – often greater than an order of magnitude! Pre-training f”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “Deep neural networks are also employed for visual reasoning where they learn and infer relationships and features of multiple objects in images [41, 58, 29].”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, methodology): “The model consists of a bidirectional LSTM encoder plus an LSTM decoder with attention, similar to the question parser in (Yi et al., 2018).”
Concept Bottleneck Models (2020, background): “More recently, deep neural networks with concept bottlenecks have re-emerged as targeted tools for solving particular tasks, e.g., Fauw et al. (2018) for retinal disease diagnosis, Yi et al. (2018) for visual question-answering, and Bucher et al. (2018) for content-based image retrieval.”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, methodology): “Please refer to Appendix G.1 for our results on the Minecraft dataset (Yi et al., 2018).”
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, methodology): “A prominent line of work [21, 67, 37, 15] makes use of the functional programs annotations that are part of the CLEVR dataset.”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “2016), but they typically are suited to reinforcement learning of actions, and involve templated language for navigation or question answering tasks, if at all (Oh et al., 2017; Yi et al., 2018).”
Learning by Abstraction: The Neural State Machine (2019, methodology): “…models heavily rely on symbolic execution, either through strong supervision of program annotations [61], non-differentiable python-based functions [87], or a collection of hand-crafted modules speciﬁcally designed for each given task [60], and consequently, they have mostly been explored in…”
Answer Them All! Toward Universal Visual Question Answering Models (2019, methodology): “Neural-Symbolic VQA (NS-VQA) [53] reports a score of 99.80% on CLEVR, but uses a question parser to allocate functional modules along with highly specialized segmentation-based CNN features.”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “…al., 1981), previous work has tried to augment text inputs to LMs with audio signals (Tsai et al., 2019) or visual perception (Bakhtin et al., 2019; Yi et al., 2018), for improved game playing (Lee et al., 2022), faster language learning (Hill et al., 2020), or better decision making in general…”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, background): “Another line of work has been extracting programs from the question for explicit reasoning with modules [5] or extracting symbols from the image to reason over them [82].”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, background): “The other aims for interpretable Visual QA via neuro-symbolic learning [50, 62, 74], developing symbolic functional programs to test machines’ reasoning capability in Visual QA.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, methodology): “They also developed an oracle model, combining the neuro-symbolic visual question-answering model (Yi et al., 2018) with the dynamics prediction model (Li et al., 2019b), showing competitive performance.”