Spatial Memory for Context Reasoning in Object Detection

By Xinlei Chen, Abhinav Kumar Gupta. 2017

173 citations recorded by the source corpus.

Modeling instance-level context and object-object relationships is extremely challenging. It requires reasoning about bounding boxes of different classes, locations etc. Above all, instance-level spatial reasoning inherently requires modeling conditional distributions on previous detections. Unfortunately, our current object detection systems do not have any memory to remember what to condition on! The state-of-the-art object detectors still detect all object in parallel followed by non-maximal suppression (NMS). While memory has been used for tasks such as captioning, they mostly use image-level memory cells without capturing the spatial layout. On the other hand, modeling object-object relationships requires spatial reasoning – not only do we need a memory to store the spatial layout, but also a effective reasoning module to extract spatial patterns. This paper presents a conceptually simple yet powerful solution – Spatial Memory Network (SMN), to model the instance-level context efficiently and effectively. Our spatial memory essentially assembles object instances back into a pseudo “image” representation that is easy to be fed into another ConvNet for object-object context reasoning. This leads to a new sequential reasoning architecture where image and memory are processed in parallel to obtain detections which update the memory again. We show our SMN direction is promising as it provides 2.2% improvement over baseline Faster RCNN on the COCO dataset with VGG161.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning (2019, methodology): “Our implementation is based on the TensorFlow implementation of the Faster RCNN framework [38], [39].”
Deep Learning for Generic Object Detection: A Survey (2018, background): “In SMN, spatial memory essentially assembles object instances back into a pseudo image representation that is easy to be fed into another CNN for object relations reasoning, leading to a new sequential reasoning architecture where image and memory are processed in parallel to obtain detections which”
Video Object Detection with an Aligned Spatial-Temporal Memory (2017, background): “Recent work that adopt deep neural networks have significantly advanced the state-of-the-art in static image object detection [16,39,17,38,37,31,9,40,7].”
Iterative Visual Reasoning Beyond Convolutions (2018, methodology): “The core of our algorithm consists of two modules: the local module, based on spatial memory [4], performs pixel-level reasoning using ConvNets.”
Context-Transformer: Tackling Object Confusion for Few-Shot Detection (2020, background): “Modeling context has been a long-term challenge for object detection (Bell et al. 2016; Chen and Gupta 2017; Kantorov et al. 2016; Mottaghi et al. 2014).”
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships (2018, background): “The first type incorporates context around object or scene-level context [3, 43, 37], and the second models object-object relationships at instance-level [18, 4, 30].”
Recent Advances in Deep Learning for Object Detection (2019, background): “However, some recent efforts [1, 127, 128, 129, 3, 59, 3, 130, 131, 106] still try to exploit contextual information.”