Flow-Guided Feature Aggregation for Video Object Detection

By Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, Yichen Wei. 2017

707 citations recorded by the source corpus.

Extending state-of-the-art object detectors from image to video is challenging. The accuracy of detection suffers from degenerated object appearances in videos, e.g., motion blur, video defocus, rare poses, etc. Existing work attempts to exploit temporal information on box level, but such methods are not trained end-to-end. We present flow-guided feature aggregation, an accurate and end-to-end learning framework for video object detection. It leverages temporal coherence on feature level instead. It improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy. Our method significantly improves upon strong singleframe baselines in ImageNet VID [33], especially for more challenging fast moving objects. Our framework is principled, and on par with the best engineered systems winning the ImageNet VID challenges 2016, without additional bells-and-whistles. The code would be released.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Do Image Classifiers Generalize Across Time? (2019, background): “A similar line of work focuses on improving object detection in videos as objects become occluded or move quickly [18, 8, 33, 30].”
Deformable ConvNets V2: More Deformable, Better Results (2018, methodology): “ that reported in [8] mainly because of a better ImageNet pretrained model and the introduction of RoIAlign in object detection. For object detection on ImageNet VID, we mainly follow the protocol in [27,43,42] for the training and inference settings. The details are presented at the end of this sec”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “We also compare to a strong baseline FGFA [60] that uses optical flow to aggregate the temporal information from 21 frames (past 10 frames and future 10 frames) for offline video detection.”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, background): “Motivated by their success in NLP tasks [2, 28, 14, 6, 26, 30, 31, 36], attention mechanisms have also been employed in computer vision applications such as relational reasoning among objects [3, 33], image captioning [41], image generation [45, 42], image recognition [20, 38, 17, 22, 46, 13], and v”
Video Instance Segmentation (2019, background): “While object identity information are often utilized for improve robustness of detection algorithms [9, 36, 33], the evaluation metric is limited to per-frame detection and does not require joint object detection and tracking.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames.”
Occlusion Aware Unsupervised Learning of Optical Flow (2017, neutral): “With the accurate optical flow prediction, one could estimate the 3D structure of a scene [18], segment moving objects based on motion cues [38], track objects in a complicated environment [11], and build important visual cues for many high level vision tasks such as video action recognition [45] an”
A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit (2021, background): “Many works in the literature use the mAP as a metric to evaluate the performance of object detection models in video sequences [92–98].”
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, methodology): “In this work, we conduct extensive analysis and present a simple, strong and versatile baseline, BasicVSR, which can serve as a back-bone with abundant ﬂexibilities in design.”
Video Object Detection with an Aligned Spatial-Temporal Memory (2017, methodology): “However, [51,50] use optical flow, which needs to be computed either externally e.”
TransTrack: Multiple-Object Tracking with Transformer (2020, methodology): “In video object detection, FGFA [49] uses optical ﬂow to warp intermediate features from the previous frames to accelerate inference.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, methodology): “Memory models have been used for many applications, including reading comprehension [51], summarization [50], tracking[69], video understanding[7], and image and video captioning [70, 74].”
Video Panoptic Segmentation (2020, methodology): “It combines several existing tasks: video object segmentation [1,3,4,27,30,35,36,39] and video object detection [9,42,43], and aims at simultaneous detection, segmentation, and tracking of instances in videos.”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, background): “For handling challenging videos such as motion blur and occlusion, optical flow [46, 39], correlation operation [15], deformable convolutional networks [3, 41] and relation networks [14, 10] are the popular technologies to propagate or align features across frames.”
EDF: Ensemble, Distill, and Fuse for Easy Video Labeling (2018, background): “Object detection in the video setting can leverage frameto-frame flow estimates [32] or correlation and regressionbased tracking [3] to detect objects more consistently across frames.”
BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection (2022, positive): “The latter reutilizes the features from the previous frame based on some special architectures like LSTM [12] for feature distillation [18, 17, 22], attention mechanism [34] for feature querying [4, 6, 7], and optical flow [8] for feature alignment [49, 48].”
Temporal-Context Enhanced Detection of Heavily Occluded Pedestrians (2020, methodology): “For instance, optical flow has been utilized to achieve feature calibration [30, 31, 38], while flow estimation may be noisy when an object is heavily occluded.”