Deep Sequential Context Networks for Action Prediction

By Yu Kong, Zhiqiang Tao, Yun Raymond Fu. 2017

173 citations recorded by the source corpus.

This paper proposes efficient and powerful deep networks for action prediction from partially observed videos containing temporally incomplete action executions. Different from after-the-fact action recognition, action prediction task requires action labels to be predicted from these partially observed videos. Our approach exploits abundant sequential context information to enrich the feature representations of partial videos. We reconstruct missing information in the features extracted from partial videos by learning from fully observed action videos. The amount of the information is temporally ordered for the purpose of modeling temporal orderings of action segments. Label information is also used to better separate the learned features of different categories. We develop a new learning formulation that enables efficient model training. Extensive experimental results on UCF101, Sports-1M and BIT datasets demonstrate that our approach remarkably outperforms state-of-the-art methods, and is up to 300x faster than these methods. Results also show that actions differ in their prediction characteristics, some actions can be correctly predicted even though only the beginning 10% portion of videos is observed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. G. Wang
3. Alex ChiChung Kot
4. Amir Shahroudy
5. Ling-yu Duan
6. Weishi Zheng
7. J. Lai
8. Jianfang Hu

In Computer Vision and Pattern Recognition

1. Jun Liu
2. G. Wang
3. Amir Shahroudy
4. Alex ChiChung Kot
5. Ling-yu Duan
6. J. Lai
7. Weishi Zheng
8. Yun Raymond Fu

In Endocrinology, Diabetes and Metabolism

1. Yu Kong
2. Yun Raymond Fu
3. Jun Liu
4. Alex ChiChung Kot
5. G. Wang
6. Amir Shahroudy
7. Weishi Zheng

Citations in the corpus

• Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Most of the existing works [1], [3], [4] focus on action prediction in wellsegmented videos, for which each video contains exactly one action instance.”
• Progressive Teacher-Student Learning for Early Action Prediction (2019, methodology): “We tackle the same problem as in [16, 24, 22], i.”
• PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction (2019, background): “Action (or behavior) prediction algorithms may take different forms such as generating future frames [20, 19, 24, 6], predicting the type of action [15, 21, 7], measuring confidence in the occurrence of an event [27, 37, 10], and forecasting the motion of objects [25, 40, 43, 1, 17, 5, 8].”
• Human Action Recognition and Prediction: A Survey (2018, methodology): “a visual surveillance system powered by action recognition [74,189,82] and prediction [176,98,99] algorithms may increase the chances of capturing a criminal on video, and reduce the risk caused by criminal actions.”
• Part-Activated Deep Reinforcement Learning for Action Prediction (2018, background): “The Lai et. al [19] and Deep SCN [17] are leading approaches on the BIT dataset, which do not mine the structural information of human.”
• SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Most of the existing works in literature [23, 27, 32] focus on action prediction in the well-segmented videos, for which each video contains only one action instance.”