Video Object Detection with an Aligned Spatial-Temporal Memory
By Fanyi Xiao, Yong Jae Lee. 2017
201 citations recorded by the source corpus.
We introduce Spatial-Temporal Memory Networks for video object detection. At its core, a novel Spatial-Temporal Memory module (STMM) serves as the recurrent computation unit to model long-term temporal appearance and motion dynamics. The STMM’s design enables full integration of pretrained backbone CNN weights, which we find to be critical for accurate detection. Furthermore, in order to tackle object motion in videos, we propose a novel MatchTrans module to align the spatial-temporal memory from frame to frame. Our method produces state-of-the-art results on the benchmark ImageNet VID dataset, and our ablative studies clearly demonstrate the contribution of our different design choices. We release our code and models at http://fanyix.cs.ucdavis.edu/project/stmn/project.html.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Alan L. Yuille
- Ming-Hsuan Yang
- Jitendra Malik
- Trevor Darrell
- Andrew Zisserman
Citations in the corpus
- Do Image Classifiers Generalize Across Time? (2019, methodology): “For the R-FCN architecture, we use the model from [30]6.”
- An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, background): “Motivated by their success in NLP tasks [2, 28, 14, 6, 26, 30, 31, 36], attention mechanisms have also been employed in computer vision applications such as relational reasoning among objects [3, 33], image captioning [41], image generation [45, 42], image recognition [20, 38, 17, 22, 46, 13], and v”
- Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames.”
- A systematic framework for natural perturbations from videos (2019, methodology): “For the R-FCN architecture, we use the model from [32].”
- Joint Object Detection and Multi-Object Tracking with Graph Neural Networks (2021, background): “To deal with the issue, video detection [41]–[43] has been investigated which uses multiple frames as inputs.”
- Temporal-Context Enhanced Detection of Heavily Occluded Pedestrians (2020, background): “Object detection in videos has been actively studied recently [50, 51, 38, 39, 40, 41, 42, 43, 30, 31, 38], exploring different ways to take advantage of temporal cues.”
- Deep Learning for Unmanned Aerial Vehicle-Based Object Detection and Tracking: A survey (2021, background): “[145] proposed a motion-aware network (MANet) to directly learn motion information over a long period of time by fusing multiple frame features.”