Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition

By Shuyang Sun, Zhanghui Kuang, Wanli Ouyang, Lu Sheng, Wayne Zhang. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

322 citations recorded by the source corpus.

Motion representation plays a vital role in human action recognition in videos. In this study, we introduce a novel compact motion representation for video action recognition, named Optical Flow guided Feature (OFF), which enables the network to distill temporal information through a fast and robust approach. The OFF is derived from the definition of optical flow and is orthogonal to the optical flow. The derivation also provides theoretical support for using the difference between two frames. By directly calculating pixel-wise spatio-temporal gradients of the deep feature maps, the OFF could be embedded in any existing CNN based video action recognition framework with only a slight additional cost. It enables the CNN to extract spatiotemporal information, especially the temporal information between frames simultaneously. This simple but powerful idea is validated by experimental results. The network with OFF fed only by RGB inputs achieves a competitive accuracy of 93.3% on UCF-101, which is comparable with the result obtained by two streams (RGB and optical flow), but is 15 times faster in speed. Experimental results also show that OFF is complementary to other motion modalities such as optical flow. When the proposed method is plugged into the state-of-the-art video action recognition framework, it has 96.0% and 74.2% accuracy on UCF-101 and HMDB-51 respectively. The code for this project is available at: https://github.com/kevin-ssy/Optical-Flow-Guided-Feature

Subfields: Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

View Paper: https://www.semanticscholar.org/paper/42517e072406ea6f8d2c579d98ee3f9918a8a1d3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Our proposed AdaMML framework is also related to prior works in joint appearance and motion modeling [43, 31, 10] that focuses on combining RGB and optical flow streams.”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “…cameras and Internet videos, action recognition task has drawn much attention in the community and comes up with great progress (Feichten-hofer, Pinz, and Zisserman 2016; Wang et al. 2016; Sun et al. 2018; Carreira and Zisserman 2017; Su, Zhao, and Lin 2018; Liu et al. 2018; Tran et al. 2015).”
Deep Spatial Gradient and Temporal Depth Learning for Face Anti-Spoofing (2020, background): “In Model 6, STSTB is replaced by normal OFF [47].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “OFF [37] and motion feature network [26] are proposed for integrating motion information modeling into a spatial CNN network, instead of using two streams.”
FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “In many downstream video tasks, such as action recognition [46,37,61], video inpainting [28,51,13], video super-resolution [30,5,39], and frame interpolation [52,33,20], optical flow serves as a fundamental component providing dense correspondences as valuable clues for prediction.”
STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, background): “Recently many approaches have been proposed to estimate optical flow with CNN [5, 14, 6, 21] or explored alternatives of optical flow [33, 39, 26, 18].”
Vision-based human activity recognition: a survey (2020, methodology): “…silhouettes [13, 56, 72], contours [72], motion history images (MHI) [6, 70], motion energy images (MEI) [6, 70] and optical flow [71, 93, 165, 190] Volume-based : These approaches represent video as spatial-temporal volume and may rely on features like texture, color, posture, histograms of…”
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition (2019, background): “Some works try to jointly optimize optical flow estimation and the classification network [36, 5], or implicitly model optical flow in RGB network [24].”
X3D: Expanding Architectures for Efficient Video Recognition (2020, unknown): “[2,5,9, 11, 13, 18, 28, 35, 38, 42, 44, 55, 56, 63, 65, 66, 72, 76, 84–86].”
Real-Time Intermediate Flow Estimation for Video Frame Interpolation (2020, background): “Optical flow estimation is a long-standing vision task that aims to estimate the per-pixel motion, useful in many downstream tasks [55,65,33,64].”
Representation Flow for Action Recognition (2018, methodology): “It clearly outperforms existing motion representation methods including TVNet [5] and OFF [21] in both speed and accuracy, which we experimentally confirm.”
MARS: Motion-Augmented RGB Stream for Action Recognition (2019, unknown): “Very recent works have attempted to model appearance and motion into a single stream [21, 35], with modules designed to better exploit temporal information, leading to complex architectures for a modest gain.”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “As shown in [13], the use of optical flow limits the application of several methods in online scenarios, pushing the community either to investigate alternative paths [27, 8] or towards single-stream architectures [63, 13, 29, 52, 44].”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, background): “There have already existed pretty much work on improving the efficiency of action recognition in videos [12, 35, 20, 2, 34, 27, 22, 28, 25, 19].”