Structure-Preserving Binary Representations for RGB-D Action Recognition

By Mengyang Yu, Li Liu, Ling Shao. 2016

116 citations recorded by the source corpus.

In this paper, we propose a novel binary local representation for RGB-D video data fusion with a structure-preserving projection. Our contribution consists of two aspects. To acquire a general feature for the video data, we convert the problem to describing the gradient fields of RGB and depth information of video sequences. With the local fluxes of the gradient fields, which include the orientation and the magnitude of the neighborhood of each point, a new kind of continuous local descriptor called Local Flux Feature(LFF) is obtained. Then the LFFs from RGB and depth channels are fused into a Hamming spacevia the Structure Preserving Projection (SPP). Specifically, an orthogonal projection matrix is applied to preserve the pairwise structure with a shape constraint to avoid the collapse of data structure in the projected space. Furthermore, a bipartite graph structure of data is taken into consideration, which is regarded as a higher level connection between samples and classes than the pairwise structure of local features. The extensive experiments show not only the high efficiency of binary codes and the effectiveness of combining LFFs from RGB-D channels via SPP on various action recognition benchmarks of RGB-D data, but also the potential power of LFF for general action recognition.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “Previous works [28, 16, 10, 58, 61, 13] showed the effectiveness of fusing the two modalities for 3D action recognition.”
Jointly Learning Heterogeneous Features for RGB-D Activity Recognition (2015, background): “Recent works also showed that the fusion of the RGB and depth sequences can largely improve the recognition of activities with object interactions [5], [15], [16], [20], [30], [41], [51], [55].”
ChaLearn looking at people: A review of events and resources (2017, background): “A total of 17 papers were published withiin this Special Issue at IEEE TPAMI [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47].”
Graph Distillation for Action Detection with Privileged Modalities (2017, background): “Specifically, with the availability of depth sensors and joint tracking algorithms, extensive research has been done on action classification and detection in RGB-D videos [39,47,48,61] as well as skeleton sequences [24,30,31,32,46,63].”