Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition

By Yansong Tang, Yi Tian, Jiwen Lu, Peiyang Li, Jie Zhou. 2018

380 citations recorded by the source corpus.

In this paper, we propose a deep progressive reinforcement learning (DPRL) method for action recognition in skeleton-based videos, which aims to distil the most informative frames and discard ambiguous frames in sequences for recognizing actions. Since the choices of selecting representative frames are multitudinous for each video, we model the frame selection as a progressive process through deep reinforcement learning, during which we progressively adjust the chosen frames by taking two important factors into account: (1) the quality of the selected frames and (2) the relationship between the selected frames to the whole video. Moreover, considering the topology of human body inherently lies in a graph-based structure, where the vertices and edges represent the hinged joints and rigid bones respectively, we employ the graph-based convolutional neural network to capture the dependency between the joints for action recognition. Our approach achieves very competitive performance on three widely used benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “For example, the frames are first sampled uniformly in [189], [190], and discrete decisions are made for each selected frame to go forward or backward step by step.”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “Speciﬁcally, many of them have been evaluated based on the preliminary version [47] of our dataset, or pre-trained on it for transfer learning for other tasks [43], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79].”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, background): “3 2018 DPRL [28] 83.”
Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition (2018, methodology): “Action recognition methods based on skeleton data have been widely investigated and attracted considerable attention due to their strong adaptability to the dynamic circumstance and complicated background [31, 8, 6, 27, 22, 29, 33, 19, 20, 21, 14, 13, 23, 18, 17, 32, 30, 34].”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, methodology): “Kinect skeletons are mapped to color images representing 3D data and used as input in [107] for a ResNet CNN. Tang et al. [108] apply reinforcement learning for a graph based CNN (GCNN) that captures structural and temporal information from 3D skeleton input.”
Reinforced Attention for Few-Shot Learning and Beyond (2021, methodology): “Reinforcement learning is easily utilized in sequential visual tasks, like tracking [64, 3, 41], action recognition [63, 52, 4], visual navigation [57, 67], video summarization [29] and person re-id [25].”
Skeleton-Based Action Recognition With Directed Graph Neural Networks (2019, methodology): “Instead of representing skeleton data as sequences or pseudo-images, graph-based methods model the data as a graph with joints as vertexes and bones as edges [34, 29, 26].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, background): “[40] focuses on skeletonbased action recognition and RL is adopted to distinguish discriminative poses.”
Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data (2021, background): “Various skeleton-based action recognition approaches [5, 29, 13, 17, 20, 18, 19, 30, 42, 15, 27, 31, 39, 14] have been proposed.”
Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition (2019, methodology): “We compare with covering hand-crafted methods [27], RNN/CNNbased deep learning models [6, 22, 20, 14, 21, 13, 18] and recent graph-based methods [29, 26, 23].”
Skeleton-Based Action Recognition With Shift Graph Convolutional Network (2020, background): “In the computer vision field, skeleton-based human action recognition has attracted much attention due to its robustness against the dynamic circumstance and complicated background [2, 12, 14, 16, 20, 21, 23–25,27, 30, 34, 36].”
Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition (2021, methodology): “GCNs have been successfully adopted to skeleton-based action recognition [20, 24, 32, 34, 36, 27] and most of them follow the feature update rule of [11].”
Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition (2019, background): “define the edges for both physically disconnected and connected joint pairs for better constructing the graph [44].”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “More recently, GCN-basedHAR has become a hot research direction [188], [189], [190], [191], [192].”
Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching (2019, methodology): “Deep STGCK (Li et al. 2018) 74.9 86.3 AAAI2018 ST-GCN (Yan et al. 2018) 81.5 88.3 AAAI2018 DPRL (Tang et al. 2018) 83.5 89.8 CVPR2018 SR-TSL (Si et al. 2018) 84.8 92.4 ECCV2018 STGR-GCN (Li et al. 2019a) 86.9 92.3 AAAI2019 GR-GCN (Gao et al. 2019) 87.5 94.3 ACMM2019 AS-GCN (Li et al. 2019b) 86 (Li…”