Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks

By Pichao Wang, Wanqing Li, Zhimin Gao, Yuyao Zhang, Chang-Fu Tang, Philip O. Ogunbona. 2017

136 citations recorded by the source corpus.

Scene flow describes the motion of 3D objects in real world and potentially could be the basis of a good feature for 3D action recognition. However, its use for action recognition, especially in the context of convolutional neural networks (ConvNets), has not been previously studied. In this paper, we propose the extraction and use of scene flow for action recognition from RGB-D data. Previous works have considered the depth and RGB modalities as separate channels and extract features for later fusion. We take a different approach and consider the modalities as one entity, thus allowing feature extraction for action recognition at the beginning. Two key questions about the use of scene flow for action recognition are addressed: how to organize the scene flow vectors and how to represent the long term dynamics of videos based on scene flow. In order to calculate the scene flow correctly on the available datasets, we propose an effective self-calibration method to align the RGB and depth data spatially without knowledge of the camera parameters. Based on the scene flow vectors, we propose a new representation, namely, Scene Flow to Action Map (SFAM), that describes several long term spatio-temporal dynamics for action recognition. We adopt a channel transform kernel to transform the scene flow vectors to an optimal color space analogous to RGB. This transformation takes better advantage of the trained ConvNets models over ImageNet. Experimental results indicate that this new representation can surpass the performance of state-of-the-art methods on two large public datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “[81] proposed a “scene flow to action map” representation for RGB+D based action recognition with CNNs.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “It has been shown that fine-tuning motion-based networks with spatial data (ImageNet) is more effective than training from scratch [56, 21, 51, 22].”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[308] extracted scene flow features from the spatially aligned and temporally synchronized RGB and depth frames.”
Joint Distance Maps Based Action Recognition With Convolutional Neural Networks (2017, background): “R ECOGNITION of human actions from Red, Green, Blue, and Depth (RGB-D) data has attracted increasing attention [1], [2] in computer vision in recent years due to the recent advance of easy-to-use and low-cost depth sensors such as Kinect sensors.”
Depth Pooling Based Large-Scale 3-D Action Recognition With Convolutional Neural Networks (2018, methodology): “4) Computational cost: we compared the computational cost of the proposed method in the test phase without any code optimization with the computational cost of the RGB+D-based methods MFSK+DeepID [58], SFAM [55], depth-based method WHDMM [40] on ChaLearn LAP IsoGD dataset.”
Spatially and Temporally Structured Global to Local Aggregation of Dynamic Depth Information for Action Recognition (2018, methodology): “Notice that SFAM [55] adopted both RGB and depth modalities while the proposed method used only depth modality for recognition.”
Global Context-Aware Attention LSTM Networks for 3D Action Recognition (2017, methodology): “Although deep learning based methods [40, 36, 55] have been used for action recognition in existing works, most of them do not focus on attention.”
Structured Images for RGB-D Action Recognition (2017, background): “For example, one can conventionally consider a video as a sequence of still images [40] with some form of temporal smoothness [27, 15] and feed them into a ConvNet, or extend ConvNet to a third, temporal dimension [16, 30] by replacing 2D filters with 3D ones, or regard the video as the output of a ”