Else-Net: Elastic Semantic Network for Continual Action Recognition from Skeleton Data

By Tianjiao Li, Qiuhong Ke, Hossein Rahmani, Rui En Ho, Henghui Ding, Jun Liu. 2021

47 citations recorded by the source corpus.

Most of the state-of-the-art action recognition methods focus on offline learning, where the samples of all types of actions need to be provided at once. Here, we address continual learning of action recognition, where various types of new actions are continuously learned over time. This task is quite challenging, owing to the catastrophic forgetting problem stemming from the discrepancies between the previously learned actions and current new actions to be learned. Therefore, we propose Else-Net, a novel Elastic Semantic Network with multiple learning blocks to learn diversified human actions over time. Specifically, our Else-Net is able to automatically search and update the most relevant learning blocks w.r.t. the current new action, or explore new blocks to store new knowledge, preserving the unmatched ones to retain the knowledge of previously learned actions and alleviates forgetting when learning new actions. Moreover, even though different human actions may vary to a large extent as a whole, their local body parts can still share many homogeneous features. Inspired by this, our proposed Else-Net mines the shared knowledge of the decomposed human body parts from different actions, which benefits continual learning of actions. Experiments show that the proposed approach enables effective continual action recognition and achieves promising performance on two large-scale action recognition datasets.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Tong Lu
6. Wenhai Wang
7. G. Wang
8. Amir Shahroudy

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Tong Lu
6. Wenhai Wang
7. Yirui Wu
8. Guangchen Shi

Citations in the corpus

• Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, background): “Incremental learning [22, 35] studies how to extend the knowledge of a model without a performance drop on old knowledge.”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[158] proposed an Elastic Semantic Network (Else-Net) for skeleton based human action recognition, which comprises a base model followed by a stack of multiple elastic units.”
• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “[243] proposed an Elastic Semantic Network (ElseNet), which consists of a GCN backbone model followed by multiple layers of elastic units for Continual HAR.”