The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

By Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Josh Tenenbaum, Jiajun Wu. 2019

861 citations recorded by the source corpus.

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

Subfields: Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “At a somewhat smaller scale Mao et al., (Mao, Gan, Kohli, Tenenbaum, & Wu, 2019) have recently proposed a hybrid neural net-symbolic system for visual question answering called NS-CL (short for the Neuro-Symbolic concept learner) that surpasses the deep learning alternatives they examined.”
Environmental drivers of systematicity and generalization in a situated agent (2019, background): “…designed to support some form of symbolic computation (such as graphs (Battaglia et al., 2018), modular-components deﬁned by symbolic parses (Andreas et al., 2016; Bahdanau et al., 2018), explicit latent variables (Higgins et al., 2017) or other neuro-symbolic hybrid methods (Mao et al., 2019)).”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “In contrast, Mao et al. [30] use element-wise min operation for intersection and max for union.”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “Another attempt to bridge the gap comes from the neurosymbolic perspective (Gupta et al., 2019; Hudson and Manning, 2019; Mao et al., 2019; Yi et al., 2018).”
Fusion of Detected Objects in Text for Visual Question Answering (2019, neutral): “The Neuro-Symbolic Concept Learner however relies on a semantic parser to intepret language, while B2T2 uses a Transformer to construct a joint representation of textual tokens and visual tokens.”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “Deep neural networks are also employed for visual reasoning where they learn and infer relationships and features of multiple objects in images [41, 58, 29].”
On the Binding Problem in Artificial Neural Networks (2020, background): “A common variant uses a neural network as a perceptual interface (or pre-processor) tasked with learning symbols from raw data, which then serve as input to a symbolic reasoning system (e.g. Mao and Gan, 2019).”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, methodology): “This constraint could be relaxed by applying unsupervised/weaklysupervised methods for scene decomposition and concept discovery (Burgess et al., 2019; Mao et al., 2019).”
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, methodology): “To create tellable systems, we see promise in approaches that combine elements of symbolic AI with successful subsymbolic approaches to allow humans and machines to operate on shared conceptualizations of theworld.(92,93) How this can best be achieved is currently a key open problem in AI.”
Neural Module Networks for Reasoning over Text (2019, background): “Recent works (Gupta & Lewis, 2018; Mao et al., 2019) also use domain-knowledge to alleviate issues in learning by using curriculum learning to train the executor first on simple questions for which parsing is not an issue.”
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data (2019, background): “Mao et al. (2019) propose a neural-symbolic architecture and discuss promising results on additional speciﬁc splits of the CLEVR data, e.g. based on object counts and program depth.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Future works could explore neurosymbolic [67] or attention-based [68] methods for better generalization to novel object-relationships.”
Learning by Abstraction: The Neural State Machine (2019, methodology): “…either through strong supervision of program annotations [61], non-differentiable python-based functions [87], or a collection of hand-crafted modules speciﬁcally designed for each given task [60], and consequently, they have mostly been explored in artiﬁcial environments such as CLEVR [44].”
Logic Tensor Networks (2020, background): “Various other loosely-coupled approached have been proposed recently such as [43], where image classification is carried out by a neural network in combination with reasoning from text data for the purpose of concept learning at a higher level of abstraction than what is normally possible with pixel”
Learning to Learn Words from Visual Scenes (2019, background): “Compositional models: Due to the diversity of the visual world, there has been extensive work in computer vision on learning compositional representations for objects and attributes [35,20,34,37,32] as well as for objects and actions [22,37,58].”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, positive): “The ideas of neural-symbolic reasoning models [12,46,65,66] offer a promising step-by-step transparent visual reasoning approach.”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, background): “Neural symbolic programs take a step further and encode a library of symbolic operations to help extrapolation (Johnson et al., 2017; Mao et al., 2019; Yi et al., 2018).”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, background): “Recently, neuro-symbolic methods [55, 38] – a combination of connectionist and symbolic approaches – have pushed these limits, achieving close to perfect scores on benchmarks like CLEVR [28, 29].”