Detect to Track and Track to Detect

By Christoph Feichtenhofer, Axel Pinz, Andrew Zisserman. 2017

594 citations recorded by the source corpus.

Recent approaches for high accuracy detection and tracking of object categories in video consist of complex multistage solutions that become more cumbersome each year. In this paper we propose a ConvNet architecture that jointly performs detection and tracking, solving the task in a simple and effective way. Our contributions are threefold: (i) we set up a ConvNet architecture for simultaneous detection and tracking, using a multi-task objective for frame-based object detection and across-frame track regression; (ii) we introduce correlation features that represent object co-occurrences across time to aid the ConvNet during tracking; and (iii) we link the frame level detections based on our across-frame tracklets to produce high accuracy detections at the video level. Our ConvNet architecture for spatiotemporal object detection is evaluated on the large-scale ImageNet VID dataset where it achieves state-of-the-art results. Our approach provides better single model performance than the winning method of the last ImageNet challenge while being conceptually much simpler. Finally, we show that by increasing the temporal stride we can dramatically increase the tracker speed.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Do Image Classifiers Generalize Across Time? (2019, background): “A similar line of work focuses on improving object detection in videos as objects become occluded or move quickly [18, 8, 33, 30].”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, methodology): “We provide ﬁne-grained, pixel-level annotations for images from each of the 10,000 video clips randomly sampled from the whole dataset.”
Video Instance Segmentation (2019, background): “While object identity information are often utilized for improve robustness of detection algorithms [9, 36, 33], the evaluation metric is limited to per-frame detection and does not require joint object detection and tracking.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames.”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, background): “D&T (Feichtenhofer et al. 2017) propose a Siamese network which takes input of adjacent frames and predicts inter-frame displacements between bounding boxes.”
Tracking Objects as Points (2020, methodology): “Recent work on simultaneous detection and tracking [1,8] has made progress in alleviating some of this complexity.”
Temporal-Context Enhanced Detection of Heavily Occluded Pedestrians (2020, methodology): “Another direction is to associate proposal or detection boxes for tube classification and detection rescoring [34, 35, 33, 37, 32].”
R-FCN-3000 at 30fps: Decoupling Detection and Classification (2017, background): “…fully-convolutional representations provide an efﬁcient [19] solu-tion for tasks like object detection [6], instance segmentation [22], tracking [10], relationship detection [41] etc., they require class-speciﬁc sets of ﬁlters for each class that pro-hibits their application for large number of…”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Other methods obtain video object proposals [25,29,35,42] based on cues such as actions [29,35] or image-level objects of interest [42,67].”
TransTrack: Multiple-Object Tracking with Transformer (2020, background): “D&T [9] uses a siamese network with the current and past frame as input and predict inter-frame offsets between bounding boxes.”
Memory Enhanced Global-Local Aggregation for Video Object Detection (2020, methodology): “The main focus of recent methods [16, 12, 37, 36, 35, 9, 27, 1, 31, 7, 30] towards solving video object detection is improving the performance of per-frame detection by exploiting information in the temporal dimension.”
Fast Online Object Tracking and Segmentation: A Unifying Approach (2018, background): “[31] considerably improve the performance of SiamFC by relying on a region proposal network (RPN) [53, 16], which allows to estimate the target location with a bounding box of variable aspect ratio.”
EDF: Ensemble, Distill, and Fuse for Easy Video Labeling (2018, background): “Object detection in the video setting can leverage frameto-frame flow estimates [32] or correlation and regressionbased tracking [3] to detect objects more consistently across frames.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, background): “, a huge number of frames required to be fast recognized, heavy occlusion, object disappearing and unconventional object-to-camera poses [18].”
A systematic framework for natural perturbations from videos (2019, background): “7% drop in [6] over the same time horizon in the video.”
StarNet: Targeted Computation for Object Detection in Point Clouds (2019, background): “Finally, we are particularly interested in studying how this system may be amenable to object tracking [3, 17, 16] as we suspect that because of the design, the computational demands may scale as the difference between successive time points as opposed to operating on the entirety of the scene [11].”
Automatic Adaptation of Object Detectors to New Domains Using Self-Training (2019, background): “We note that improvements in network architecture specific to video object recognition [12, 59] are orthogonal to our current motivation.”