A dataset and architecture for visual reasoning with a working memory

By Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, David Sussillo. 2018

56 citations recorded by the source corpus.

A vexing problem in artificial intelligence is reasoning about events that occur in complex, changing visual stimuli such as in video analysis or game play. Inspired by a rich tradition of visual reasoning and memory in cognitive psychology and neuroscience, we developed an artificial, configurable visual question and answer dataset (COG) to parallel experiments in humans and animals. COG is much simpler than the general problem of video analysis, yet it addresses many of the problems relating to visual and logical reasoning and memory -- problems that remain challenging for modern deep learning architectures. We additionally propose a deep learning architecture that performs competitively on other diagnostic VQA datasets (i.e. CLEVR) as well as easy settings of the COG dataset. However, several settings of COG result in datasets that are progressively more challenging to learn. After training, the network can zero-shot generalize to many new tasks. Preliminary analyses of the network architectures trained on COG demonstrate that the network accomplishes the task in a manner interpretable to humans.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Specifically, MarioQA [28] and COG [40] generate synthetic questions with simulated environments.”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “Recently, the relatively simple language and reasoning in existing resources motivated datasets that focus on compositional language, mostly using synthetic data for language and vision (Andreas et al., 2016; Johnson et al., 2017a; Kuhnle and Copestake, 2017; Kahou et al., 2018; Yang et al., 2018).”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, neutral): “COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments to generate synthetic data and controllable reasoning tasks.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “The goal of COG is to address problems relating to visual and logical reasoning and memory.”
Task representations in neural networks trained to perform many cognitive tasks (2019, background): “Indeed, our network is not able to perform the DMS task using composite rule inputs; more broadly, it remains unclear whether standard modern recurrent network architectures can accomplish challenging compositional tasks 39,40 .”
Artificial Neural Networks for Neuroscientists: A Primer (2020, background): “Such top-down spatial and feature attention can be included in convolutional neural networks (Lindsay and Miller, 2018; Yang et al., 2018).”
Synthetic Data for Deep Learning (2019, background): “We also note a recently published COG dataset produced by Google Brain researchers [667] that extends CLEVR’s ideas to video processing.”