When Video Classification Meets Incremental Classes

By Hanbin Zhao, Xin Qin, Shihao Su, Zibo Lin, Xi Li. 2021

32 citations recorded by the source corpus.

With the rapid development of social media, tremendous videos with new classes are generated daily, which raise an urgent demand for video classification methods that can continuously update new classes while maintaining the knowledge of old videos with limited storage and computing resources. In this paper, we summarize this task as Class-Incremental Video Classification (CIVC) and propose a novel framework to address it. As a subarea of incremental learning tasks, the challenge of catastrophic forgetting is unavoidable in CIVC. To better alleviate it, we utilize some characteristics of videos. First, we decompose the spatio-temporal knowledge before distillation rather than treating it as a whole in the knowledge transfer process; trajectory is also used to refine the decomposition. Second, we propose a dual granularity exemplar selection method to select and store representative video instances of old classes and key-frames inside videos under a tight storage budget. We benchmark our method and previous SOTA class-incremental learning methods on Something-Something V2 and Kinetics datasets, and our method outperforms previous methods significantly.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Ziwei Liu
5. Qiwen Wang
6. Bernard Ghanem
7. Zhiyuan Qi
8. Lanqing Hong

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Luc Van Gool
4. Ziwei Liu
5. Bernard Ghanem
6. Dacheng Tao
7. Bernt Schiele
8. Ming-Hsuan Yang

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Qiwen Wang
5. Ziwei Liu
6. Zhiyuan Qi
7. Lanqing Hong
8. Zhi-Hong Qi

Citations in the corpus

• Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “Existing literature focuses mostly on supervised continual learning [50,53,41], which mainly falls into the following four groups.”
• vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022, background): “Remarkably, few research efforts have addressed continual learning with video [24, 26, 44].”
• MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, neutral): “Recently, there has been a large body of research in incremental learning [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43].”
• Deep Class-Incremental Learning: A Survey (2023, background): “CVIC [162] decouples the distillation term into spatial and temporal features for video classification.”
• Online Continual Learning For Interactive Instruction Following Agents (2024, background): “Recently, continual learning has also been actively investigated in more challenging task setups such as video domains (Zhao et al., 2021b; Villa et al., 2022; Park et al., 2021).”
• Class-Incremental Learning: A Survey (2023, background): “CVIC [191] decouples the distillation term into spatial and temporal features for video classification.”