Object Detection in Video with Spatiotemporal Sampling Networks

By Gedas Bertasius, Lorenzo Torresani, Jianbo Shi. 2018

240 citations recorded by the source corpus.

We propose a Spatiotemporal Sampling Network (STSN) that uses deformable convolutions across time for object detection in videos. Our STSN performs object detection in a video frame by learning to spatially sample features from the adjacent frames. This naturally renders the approach robust to occlusion or motion blur in individual frames. Our framework does not require additional supervision, as it optimizes sampling locations directly with respect to object detection performance. Our STSN outperforms the state-of-the-art on the ImageNet VID dataset and compared to prior video object detection methods it uses a simpler design, and does not require optical flow data for training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “To solve this issue, following (Bertasius et al., 2018; Dosovitskiy et al., 2015), we present an easy plug-in called temporal feature calibration as illustrated in Fig.”
Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving (2020, background): “Previous works on video object detection [11, 14, 3, 20] has demonstrated that exploring spatial and temporal correlations is useful for learning an accurate representation for the current frame and achieving better object detection performance.”
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution (2018, methodology): “It has been applied to address several high-level vision tasks such as object detection [4], video object detection [2], semantic seg-𝐼 mentation [4], and human pose estimation [24].”
A Survey of Deep Learning-Based Object Detection (2019, background): “Video detectors [266][267][268][269][270][271][272][273][274][275] exploit temporal contexts to meet this challenge.”
Memory Enhanced Global-Local Aggregation for Video Object Detection (2020, methodology): “FGFA [36] ResNet-101 78.4 ST-Lattice [4] ResNet-101 79.6 MANet [27] ResNet-101 80.3 D&T [9] ResNet-101 80.2 STSN [1] ResNet-101+DCN 80.4 STMN [31] ResNet-101 80.5 SELSA [30] ResNet-101 80.5 OGEMN [6] ResNet-101+DCN 81.6 RDN [7] ResNet-101 83.8 MEGA (ours) ResNet-101 84.5 FGFA [36] Inception-ResNet 8”
Crossover Learning for Fast Online Video Instance Segmentation (2021, unknown): “MaskProp [2] is a state-of-the-art offline VIS approach that proposes a novel mask propagation mechanism in combination with Spatiotemporal Sampling Network [3], Hy-”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “To achieve this goal, we perform an efﬁcient temporal propagation via a single deformable convolution [12], which has been used for temporally aligning features in previous works [4, 3, 13].”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, background): “For handling challenging videos such as motion blur and occlusion, optical flow [46, 39], correlation operation [15], deformable convolutional networks [3, 41] and relation networks [14, 10] are the popular technologies to propagate or align features across frames.”
EDVR: Video Restoration With Enhanced Deformable Convolutional Networks (2019, methodology): “Deformable convolutions are widely used in various tasks such as video object detection [1], action recognition [52], semantic segmentation [3], and video super-resolution [40].”