Early Action Prediction by Soft Regression

By Jianfang Hu, Weishi Zheng, Lianyang Ma, G. Wang, J. Lai, Jianguo Zhang. 2019

117 citations recorded by the source corpus.

We propose a novel approach for predicting on-going action with the assistance of a low-cost depth camera. Our approach introduces a soft regression-based early prediction framework. In this framework, we estimate soft labels for the subsequences at different progress levels, jointly learned with an action predictor. Our formulation of soft regression framework 1) overcomes a usual assumption in existing early action prediction systems that the progress level of on-going sequence is given in the testing stage; and 2) presents a theoretical framework to better resolve the ambiguity and uncertainty of subsequences at early performing stage. The proposed soft regression framework is further enhanced in order to take the relationships among subsequences and the discrepancy of soft labels over different classes into consideration, so that a Multiple Soft labels Recurrent Neural Network (MSRNN) is finally developed. For real-time performance, we also introduce a new RGB-D feature called “local accumulative frame feature (LAFF)”, which can be computed efficiently by constructing an integral feature map. Our experiments on three RGB-D benchmark datasets and an unconstrained RGB action set demonstrate that the proposed regression-based early action prediction model outperforms existing models significantly and also show that the early action prediction on RGB-D sequence is more accurate than that on RGB channel.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Alex ChiChung Kot
4. Ling-yu Duan
5. Amir Shahroudy
6. J. Lai
7. Weishi Zheng
8. Hossein Rahmani

Citations in the corpus

• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, methodology): “The methods in [30], [47], [48], [94], [102], [105] all use RNN/LSTM for 3D action recognition, and the approaches in [49], [103], [104], [106], [107] all use convolutional networks for spatial and temporal modeling.”
• Progressive Teacher-Student Learning for Early Action Prediction (2019, methodology): “[16] proposed a soft regression framework to learn a robust action predictor from both the partial videos and full videos.”