Iterative Visual Reasoning Beyond Convolutions

By Xinlei Chen, Li-Jia Li, Li Fei-Fei, Abhinav Kumar Gupta. 2018

225 citations recorded by the source corpus.

We present a novel framework for iterative visual reasoning. Our framework goes beyond current recognition systems that lack the capability to reason beyond stack of convolutions. The framework consists of two core modules: a local module that uses spatial memory [4] to store previous beliefs with parallel updates; and a global graph-reasoning module. Our graph module has three components: a) a knowledge graph where we represent classes as nodes and build edges to encode different types of semantic relationships between them; b) a region graph of the current image where regions in the image are nodes and spatial relationships between these regions are edges; c) an assignment graph that assigns regions to classes. Both the local module and the global module roll-out iteratively and cross-feed predictions to each other to refine estimates. The final predictions are made by combining the best of both modules with an attention mechanism. We show strong performance over plain ConvNets, e.g. achieving an 8.4% absolute improvement on ADE [55] measured by per-class average precision. Analysis also shows that the framework is resilient to missing regions for reasoning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Relational inductive biases, deep learning, and graph networks (2018, methodology): “…segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D meshes and point clouds (Wang et al., 2018d), to classify regions in images (Chen et al., 2018a), to perform semi-supervised text classiﬁcation (Kipf and Welling, 2017), and in machine translation (Vaswani et al., 2017; Shaw…”
Relational Deep Reinforcement Learning (2018, background): “, [32]), and hierarchical RL and planning [33, 34] to allow structured representations and reasoning to translate more fully into structured behaviors.”
Visual Semantic Reasoning for Image-Text Matching (2019, methodology): “Inspired by recent advances in deep learning based visual reasoning [2, 35, 42], we build up a region relationship reasoning model to enhance the region-based representation by considering the semantic correlation between image regions.”
LinkNet: Relational Embedding for Scene Graph (2018, unknown): “[4] uses a graph to encode spatial and semantic relations between regions and classes and passes information among them.”
Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification (2019, methodology): “For Visual Genome (VG) [27], we use the synsets [52] instead of the raw names of the categories due to inconsistent label annotations, following [22, 12, 23].”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “…either by retrieving a set of relevant facts from the knowledge base and feeding as additional features to the model [18,7,22], or by employing a graph-based message propagation process to embed the structure of the knowledge graph within the intermediate representations of the model [3,9,2,28,14].”
Classification-by-Components: Probabilistic Modeling of Reasoning over a Set of Components (2019, methodology): “In general, the reasoning process has slight similarities to ideas mentioned in [19] and the modeling of knowledge via graph structures [20–22].”
Language-Conditioned Graph Networks for Relational Reasoning (2019, methodology): “Graph networks have been successfully applied to various tasks, from object detection [25] and region clas-siﬁcation [7] to human-object interaction [30] and activity recognition [12].”
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA (2020, methodology): “Knowledge graphs have also found extensive use in visual classification and detection [52, 13], zero-shot classification [76] and image retrieval [31].”
When Radiology Report Generation Meets Knowledge Graph (2020, methodology): “As many existing works (Yao et al. 2018; Liang et al. 2018; Chen et al. 2018; Norcliffe-Brown, Vafeias, and Parisot 2018; Hu et al. 2019), we used the graph convolution as a means of message passing and node interaction.”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, background): “To build contextualized representations for objects in a visual scene, recent studies have introduced graph networks to iteratively pass messages among linked object nodes [9, 18].”
Deep reinforcement learning with relational inductive biases (2018, background): “Other future directions include exploring perceiving complex scenes via more structured formats, such as scene graphs (Xu et al., 2017; Chen et al., 2018), which could be powerful additions to our approach’s input module.”
Symbolic Graph Reasoning Meets Convolutions (2018, background): “[7] leverage local region-based reasoning and global reasoning to facilitate object detection.”
MUREL: Multimodal Relational Reasoning for Visual Question Answering (2019, background): “Recently, the deep learning community started to tackle complex visual reasoning problems such as relationship detection [26], object recognition [9], abstract reasoning [33] or visual causality [25], while more theoretical work attempt to formalize relational reasoning [7].”
Bayesian Relational Memory for Semantic Visual Navigation (2019, background): “Similar ideas of using semantic knowledge to enhance reasoning have been applied to image classification [34, 56], segmentation [52, 64], situation recognition [33], visual question answering [57, 9, 53, 28, 26], image retrieval [29] and relation detection [62].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “A comparison of the chains [68], fully connected graph [21], [23], [27], [31], [81], [82], [83], subgraph [27] and dynamic tree [72] structure of the scene graph.”
Learning to Compose Dynamic Tree Structures for Visual Contexts (2018, background): “As shown in Figure 1, two popular structures are chains [57] and fullyconnected graphs [7, 10, 15, 25, 50, 55, 49], where the context is encoded by sequential models such as bidirectional LSTM [19] for chains and CRF-RNN [61] for graphs.”