Object Detection from Video Tubelets with Convolutional Neural Networks

By Kai Kang, Wanli Ouyang, Hongsheng Li, Xiaogang Wang. Computer Vision and Pattern Recognition, 2016

389 citations recorded by the source corpus.

Deep Convolution Neural Networks (CNNs) have shown impressive performance in various vision tasks such as image classification, object detection and semantic segmentation. For object detection, particularly in still images, the performance has been significantly increased last year thanks to powerful deep networks (e.g. GoogleNet) and detection frameworks (e.g. Regions with CNN features (RCNN)). The lately introduced ImageNet [6] task on object detection from video (VID) brings the object detection task into the video domain, in which objects' locations at each frame are required to be annotated with bounding boxes. In this work, we introduce a complete framework for the VID task based on still-image object detection and general object tracking. Their relations and contributions in the VID task are thoroughly studied and evaluated. In addition, a temporal convolution network is proposed to incorporate temporal information to regularize the detection results and shows its effectiveness for the task. Code is available at https://github.com/ myfavouritekk/vdetlib.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1c6e067098fa86ee3f96365f28669b06f9ce0c7a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Discover and Learn New Objects from Documentaries (2017, background): “[17] to generate candidate tracklets, without performing the actual detection task.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “On the other hand, as deep learning technology has developed, the corresponding object detection research has become increasingly mature [16], [62], [63], [64], [65], [66].”
Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification (2017, background): “Multi-label image classification is an important task in computer vision with various applications, such as scene recognition [4, 30, 31], multi-object recognition [25, 19, 18], human attribute recognition [24], etc.”
ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification (2017, background): “Action recognition is a well studied problem with standard datasets [4, 8, 18, 28, 41, 44] focused on tasks such as classification [15, 42, 53, 58, 59] and temporal or spatiotemporal localization [25, 60, 62].”
Factors in Finetuning Deep Model for Object Detection with Long-Tail Distribution (2016, background): “Deep learning is applied for object detection in many works [12, 30, 18, 14, 35, 47, 43, 11, 28, 17, 27, 25, 26, 45, 15].”
Person Search with Natural Language Description (2017, methodology): “Different from convolutional neural network which works well in image classiﬁcation [20, 10] and object detection [17, 16], recurrent neural network is more suitable in processing sequential data.”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, result): “Similar ideas have also been explored in [40], [41], [42], [43], [44].”
Video Visual Relation Detection (2017, methodology): “Video object detection aims to detect objects belonging to the pre-de ned categories and localize them with bounding box trajectories in a given video [11].”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, methodology): “Box-level methods [18, 11, 49] combine image object detection and tracking.”
Temporal-Context Enhanced Detection of Heavily Occluded Pedestrians (2020, methodology): “Another direction is to associate proposal or detection boxes for tube classification and detection rescoring [34, 35, 33, 37, 32].”
Temporal Context Network for Activity Localization in Videos (2017, background): “A temporal Convolutional Neural Network [16] enforces temporal consistency and obtains consistent performance improvements over still-image detections.”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, background): “For the human visual perception system, temporal information plays a crucial role in inferring the motion state of objects and identifying occluded objects, and many works in vision fields have demonstrated the effectiveness of using video data [2, 27, 26, 33, 19].”
Detect to Track and Track to Detect (2017, methodology): “In [18] tubelet proposals are generated by applying a tracker to frame-based bounding box proposals.”
Recent Advances in Deep Learning for Object Detection (2019, methodology): “CNN) [53] which was extended from Faster RCNN and incorpo1751”
Flow-Guided Feature Aggregation for Video Object Detection (2017, methodology): “Such temporal information is exploited in existing video object detection methods [18, 19, 12, 23] in a simple way.”
Memory Enhanced Global-Local Aggregation for Video Object Detection (2020, methodology): “The main focus of recent methods [16, 12, 37, 36, 35, 9, 27, 1, 31, 7, 30] towards solving video object detection is improving the performance of per-frame detection by exploiting information in the temporal dimension.”
Object Detection in Videos with Tubelet Proposal Networks (2017, methodology): “Existing methods on object detection in videos generates tubelet proposals utilizing either generic single-object tracker starting at a few key frames [14] or data association methods ( i.e . tracking-by-detection methods) on per-frame object detection results [7].”