Learning by Abstraction: The Neural State Machine

By Drew A. Hudson, Christopher D. Manning. 2019

286 citations recorded by the source corpus.

We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an abstract latent space, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasets that involve compositionality, multi-step inference and diverse reasoning skills, achieving state-of-the-art results in both cases. We provide further experiments that illustrate the model's strong generalization capacity across multiple dimensions, including novel compositions of concepts, changes in the answer distribution, and unseen linguistic structures, demonstrating the qualities and efficacy of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “On GQA, neural state machine (NSM) [12] relies on a strong structural prior, which can also be incorporated into Oscar for improvement in the future.”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “Another attempt to bridge the gap comes from the neurosymbolic perspective (Gupta et al., 2019; Hudson and Manning, 2019; Mao et al., 2019; Yi et al., 2018).”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “LGCN [20] 55.8 LXMERT [51] 60.0 NSM [24] 63.0 CRF [39] 72. fully Pythonic implementation is compatibility with a wide range of existing tools, such as PyTorch JIT [42].”
VinVL: Making Visual Representations Matter in Vision-Language Models (2021, methodology): “Our results show that the object-centric representations produced by the new OD model significantly improve the performance across all the VL tasks, often by a large margin over strong baselines using the classical OD model [2], creating new state of the arts on all these tasks, including GQA on whi”
Unbiased Scene Graph Generation From Biased Training (2020, neutral): “Although this is partly due to the research gap in graph reasoning [2, 49, 16], the crux lies in the biased relationship prediction.”
KB-VLP: Knowledge Based Vision and Language Pretraining (2021, background): “Dev Test-std Dev Test-P Dev Test-std R@1 R@5 R@10 ACC-full NSM (Drew A. Hudson, 2019) – – – – – 63.17 – – – – ViLBERT (Lu et al., 2019a) 70.63 70.92 – – – – – – – – VL-BERT (Su et al., 2020) 70.50 70.83 – – – – – – – – VisualBERT (Li et al., 2019) 70.80 71.00 67.40 67.00 – – – – – – LXMERT (Tan &…”
VQA With No Questions-Answers Training (2018, background): “Many recent works focused on improving the image-question fused features [17, 9, 84, 10, 16], attention mechanisms for selecting important features [80, 54, 47, 4, 12, 59, 50, 27], including self and guided attention [83, 21, 82], applying pre-trained networks [65, 46, 86], and incorporating outputs”
Under review as a conference paper at ICLR 2020 completely free-form sampling in image space : Spatial Transformers (2019, background): “Humans are remarkably adept at making abstractions of the world (Hudson & Manning, 2019); we see in raw visual signals, abstract semantics away from deformation, and form concepts.”
Coarse-to-Fine Reasoning for Visual Question Answering (2021, methodology): “…into three groups: joint learning mechanisms: BAN [26], Pythia [24], DFAF [15], fPMC [20], STL [54], CTI [11], and MCAN [59]; reasoning-based methods: Murel [4], Re-GAT [30], MMN [6], NMS [22], and HAN [25]; and large-scale visual-language modeling: LXMERT [47], OSCAR [32], and UNITER [7].”
Extending Machine Language Models toward Human-Level Language Understanding (2019, background): “This is advocated in (59), but natural structure exhibits flexible embedding relationships and is often only approximately characterized by explicit taxonomies, motivating use of emergent connection-based rather than hard-coded representational structures (60).”
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models (2021, unknown): “…pair (including the categories and bounding boxes) in an image, predicate classiﬁcation aims to classify the relation into a relation set P , providing structured image representations that can facilitate many cross-modal tasks (Johnson et al., 2015; Hudson & Manning, 2019; Shi et al., 2019).”
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models (2020, background): “For thorough evaluation, we compare model performance on the following set of competing methods: • SOTA (task-specific state of the art): Cycle Consis- tency+BAN [56] for VQA-Rephrasings, LOL [18] for VQA-LOL Compose and Supplement, Pythia [54, 28] for VQA-Introspect, NSM [26] for GQA, SAAA [2, 30] ”
SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels (2021, unknown): “0 (Antol et al., 2015), GQA balanced version (Hudson and Manning, 2019b) and VG-QA (Zhu et al., 2016).”
MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond (2021, methodology): “Since CLEVR is synthetic, we also initiate an exploration of MoVie on the recent natural-image reasoning dataset, GQA (Hudson & Manning, 2019a).”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, methodology): “While the Neural State Machine (NSM) model [26] also builds a scene graph from the image similar to our approach, it requires converting the question text into a sequence of instructions for graph traversal.”
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, methodology): “The inferred SG can be used directly for downstream tasks such as VQA [42, 15], image captioning [40, 12] or retrieval [18, 4, 32].”