Representation Flow for Action Recognition

By A. Piergiovanni, Michael S. Ryoo. 2018

165 citations recorded by the source corpus.

In this paper, we propose a convolutional layer inspired by optical flow algorithms to learn motion representations. Our representation flow layer is a fully-differentiable layer designed to capture the `flow' of any representation channel within a convolutional neural network for action recognition. Its parameters for iterative flow optimization are learned in an end-to-end fashion together with the other CNN model parameters, maximizing the action recognition performance. Furthermore, we newly introduce the concept of learning `flow of flow' representations by stacking multiple representation flow layers. We conducted extensive experimental evaluations, confirming its advantages over previous recognition models using traditional optical flows in both computational speed and performance. The code is publicly available.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Hongsheng Li
5. Zehua Sun
6. Jifeng Dai
7. Hongwei Qin
8. G. Wang

In Biomedical Engineering

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Jifeng Dai
6. Hongsheng Li
7. G. Wang
8. Amir Shahroudy

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Hossein Rahmani
3. Qiuhong Ke
4. Zehua Sun
5. G. Wang
6. Hongsheng Li
7. Xiaogang Wang
8. Luc Van Gool

Citations in the corpus

• Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, neutral): “FcF [23] Kinetics-400 81.”
• X3D: Expanding Architectures for Efficient Video Recognition (2020, unknown): “[2,5,9, 11, 13, 18, 28, 35, 38, 42, 44, 55, 56, 63, 65, 66, 72, 76, 84–86].”
• Human Action Recognition From Various Data Modalities: A Review (2020, background): “S: Skeleton, D: Depth, IR: Infrared, PC: Point Cloud, Ac: Acceleration, Gyr: Gyroscope and Ryoo [61] proposed a trainable flow layer which captures the motion information without the need of computing optical flows.”
• FlowFormer: A Transformer Architecture for Optical Flow (2022, background): “In many downstream video tasks, such as action recognition [46,37,61], video inpainting [28,51,13], video super-resolution [30,5,39], and frame interpolation [52,33,20], optical flow serves as a fundamental component providing dense correspondences as valuable clues for prediction.”