Deep Feature Flow for Video Recognition

By Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, Yichen Wei. 2016

657 citations recorded by the source corpus.

Deep convolutional neutral networks have achieved great success on image recognition tasks. Yet, it is non-trivial to transfer the state-of-the-art image recognition networks to videos as per-frame evaluation is too slow and unaffordable. We present deep feature flow, a fast and accurate framework for video recognition. It runs the expensive convolutional sub-network only on sparse key frames and propagates their deep feature maps to other frames via a flow field. It achieves significant speedup as flow computation is relatively fast. The end-to-end training of the whole architecture significantly boosts the recognition accuracy. Deep feature flow is flexible and general. It is validated on two recent large scale video datasets. It makes a large step towards practical video recognition. Code would be released.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deformable ConvNets V2: More Deformable, Better Results (2018, methodology): “For object detection on ImageNet VID, we mainly follow the protocol in [27, 43, 42] for the training and inference settings.”
Semantic Flow for Fast and Accurate Scene Parsing (2020, methodology): “The concept of Semantic Flow is inspired from optical flow, which is widely used in video processing task [70] to represent the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by relative motion.”
Improving Semantic Segmentation via Decoupled Body and Edge Supervision (2020, background): “The sampling mechanism, proposed in the spatial transformer networks [50,51], linearly interpolates the values of the four nearest neighbor pixel of pl.”
BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies (2021, methodology): “Deep Feature Flow [51], NetWarp [8], GRFP [30], Awan and Shin [2], and Paul et al. [34] warp representations between frames using optical flow, with Accel [17] introducing a lightweight second branch to fine tune representations.”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, background): “A few recent approaches attempt to reduce computation cost in videos by exploring similarities among adjacent frames [39, 17], while our goal is to selectively choose relevant frames based on inputs.”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “For each sample at timestamp t, we randomly sample another 3 samples from the consecutive sequence of the past 2 seconds, and this random sampling strategy can augment the diversity of ego-motion [56].”
Temporal-Context Enhanced Detection of Heavily Occluded Pedestrians (2020, methodology): “For instance, optical flow has been utilized to achieve feature calibration [30, 31, 38], while flow estimation may be noisy when an object is heavily occluded.”
Online Model Distillation for Efficient Video Inference (2018, methodology): “Table 2 shows two baselines that propagate pixel segmentations using Dense Feature Flow [48], although we upgrade the flow estimation network from FlowNet2 [18] to modern methods.”
Video Panoptic Segmentation (2020, methodology): “tation (VSS), several approaches have been proposed in the literature [14, 21, 26, 33, 43].”
Coherent Online Video Style Transfer (2017, methodology): “[45] recently integrate the FlowNet [17] with image recognition networks and train the network end-to-end for fast video recognition.”
Detect to Track and Track to Detect (2017, methodology): “In deep feature flow [40] a recognition ConvNet is applied to key frames only and an optical flow ConvNet is used for propagating the deep feature maps via a flow field to the rest of the frames.”
Dynamic Video Segmentation Network (2018, methodology): “In the past decade, semantic video segmentation focusing on reducing the inference time has received little attention [29, 30].”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, background): “[117] proposes to run a convolutional sub-network only on sparse keyframes and propagate the deep feature maps to other frames via flow fields.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “• Video Segmentation: The research for VSS mainly focuses on better spatial-temporal fusion [90] or acceleration using extra cues [91], [92] in the video.”
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (2020, background): “The information-refill mechanism in IconVSR is reminiscent of the concept of interval-based processing [4, 15, 26, 35, 36, 38, 39].”
Memory Enhanced Global-Local Aggregation for Video Object Detection (2020, methodology): “The main focus of recent methods [16, 12, 37, 36, 35, 9, 27, 1, 31, 7, 30] towards solving video object detection is improving the performance of per-frame detection by exploiting information in the temporal dimension.”