Object Detection in Videos with Tubelet Proposal Networks

By Kai Kang, Hongsheng Li, Tong Xiao, Wanli Ouyang, Junjie Yan, Xihui Liu, Xiaogang Wang. 2017

198 citations recorded by the source corpus.

Object detection in videos has drawn increasing attention recently with the introduction of the large-scale ImageNet VID dataset. Different from object detection in static images, temporal information in videos is vital for object detection. To fully utilize temporal information, state-of-the-art methods [15, 14] are based on spatiotemporal tubelets, which are essentially sequences of associated bounding boxes across time. However, the existing methods have major limitations in generating tubelets in terms of quality and efficiency. Motion-based [14] methods are able to obtain dense tubelets efficiently, but the lengths are generally only several frames, which is not optimal for incorporating long-term temporal information. Appearance-based [15] methods, usually involving generic object tracking, could generate long tubelets, but are usually computationally expensive. In this work, we propose a framework for object detection in videos, which consists of a novel tubelet proposal network to efficiently generate spatiotemporal proposals, and a Long Short-term Memory (LSTM) network that incorporates temporal information from tubelet proposals for achieving high object detection accuracy in videos. Experiments on the large-scale ImageNet VID dataset demonstrate the effectiveness of the proposed framework for object detection in videos.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Do Image Classifiers Generalize Across Time? (2019, background): “A similar line of work focuses on improving object detection in videos as objects become occluded or move quickly [18, 8, 33, 30].”
Video Object Detection with an Aligned Spatial-Temporal Memory (2017, methodology): “While the Tubelet Proposal Network [24] does model long-term dependencies, it uses vectors to represent the memory of the recurrent unit, and hence loses spatial information.”
Learning Deep Neural Networks for Vehicle Re-ID with Visual-spatio-Temporal Path Proposals (2017, methodology): “For sequential data, the family of Recurrent Neural Networks including LongShort Term Memory Network(LSTM) [14] and Gated Recurrent Neural Network [4] have achieved great success for tasks including image captioning [18], speech recognition [11], visual question answering [2], person search [24], im”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Other methods obtain video object proposals [25,29,35,42] based on cues such as actions [29,35] or image-level objects of interest [42,67].”
Tracking Objects as Points (2020, background): “[16, 17] feed stacked consecutive frames into the network and do detection for a whole video segment.”
T-CNN: Tubelets With Convolutional Neural Networks for Object Detection From Videos (2016, background): “[22] proposed a tubelet proposal network to efficiently generates hundreds of tubelet proposals simultaneously.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “On the other hand, as deep learning technology has developed, the corresponding object detection research has become increasingly mature [16], [62], [63], [64], [65], [66].”
Person Search with Natural Language Description (2017, background): “be contents of images for text-image joint embedding. Deep language models for vision. Different from convolutional neural network which works well in image classiﬁcation [21,10] and object detection [18,17,16], recurrent neural network is more suitable in processing sequential data. A large number ”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, result): “Similar ideas have also been explored in [40], [41], [42], [43], [44].”
Detect to Track and Track to Detect (2017, methodology): “A more recent work [16] introduces a tube-let proposal network that regresses static object proposals over multiple frames, extracts features by applying Faster R-CNN which are ﬁnally processed by an encoder-decoder LSTM.”
A systematic framework for natural perturbations from videos (2019, methodology): “The ImageNet-Vid-Robust dataset is sourced from videos contained in the ImageNet-Vid dataset [12], we provide more details about ImageNet-Vid in the supplementary.”
Tracking Without Bells and Whistles (2019, background): “Both [28] and [27] generate multiframe bounding box tuplet proposals and extract detection scores and features with a CNN and LSTM, respectively.”