ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning

By Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, Jun Zhu. 2021

114 citations recorded by the source corpus.

Continual learning usually assumes the incoming data are fully labeled, which might not be applicable in real applications. In this work, we consider semi-supervised continual learning (SSCL) that incrementally learns from partially labeled data. Observing that existing continual learning methods lack the ability to continually exploit the unlabeled data, we propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN), which continually passes the learned data distribution to the classifier. In particular, ORDisCo replays data sampled from the conditional generator to the classifier in an online manner, exploiting unlabeled data in a time- and storage-efficient way. Further, to explicitly overcome the catastrophic forgetting of unlabeled data, we selectively stabilize parameters of the discriminator that are important for discriminating the pairs of old unlabeled data and their pseudo-labels predicted by the classifier. We extensively evaluate ORDisCo on various semi-supervised learning benchmark datasets for SSCL, and show that ORDisCo achieves significant performance improvement on SVHN, CIFAR10 and Tiny-ImageNet, compared to strong baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Jun Zhu
5. Joost van de Weijer
6. Hang Su
7. Liyuan Wang
8. Xingxing Zhang

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jun Zhu
4. Jiashi Feng
5. Luc Van Gool
6. Joost van de Weijer
7. Dacheng Tao
8. Ziwei Liu

In Meta & Continual Learning

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Liyuan Wang
5. Xingxing Zhang
6. Jun Zhu
7. Joost van de Weijer
8. Hang Su

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “Weight regularization [314], [374], [427], [428] and experience replay [157], [428] have been shown to be effective in mitigating catastrophic forgetting of generative models.”
• Towards Label-Efficient Incremental Learning: A Survey (2023, background): “Best viewed in color. dant unlabeled data to reduce label-supervision via pseudo supervision [ Wang et al. , 2021 ] .”
• Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, background): “Traditional continual learning methods mostly focus on the issue of stabilityplasticity dilemma [18,45], where a model is prone to forgetting the previously learned knowledge when adapting to new data.”
• Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, neutral): “SETS is completely different from CL and SSCL in several aspects: (i) our SETS focuses on a large-scale task sequence (more than 72K), while CL and SSCL are not scalable to this large-scale setting; (ii) our goal is to enable the model to generalize to the unseen tasks from all the previous task dis”
• CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “Representative strategies include weight regularization [20,52,1], memory replay [36,44,46], parameter isolation [41,18] and dynamic architecture [14,50].”
• Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, background): “is impractical to maintain the training data of each task due to privacy concerns and so forth [4, 11, 20, 22, 29, 30, 32].”
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Therefore, it is necessary to enable the model to continuously learn new categories, namely class-incremental learning [34,44,49].”
• Memory Replay with Data Compression for Continual Learning (2022, positive): “…performance in class-incremental learning (Liu et al., 2021a; Hu et al., 2021) and in numerous other continual learning scenarios, such as audio tasks (Ehret et al., 2020), few-shot (Tao et al., 2020b), semi-supervised (Wang et al., 2021a), and unsupervised continual learning (Khare et al., 2021).”
• AFEC: Active Forgetting of Negative Transfer in Continual Learning (2021, background): “Representative approaches include: weight regularization [12, 1, 36, 2], which selectively penalizes changes of the previously-learned parameters; parameter isolation [24, 10], which allocates a dedicated parameter subspace for each task; and memory replay [21, 9, 31, 32], which approximates and rec”
• How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2021, methodology): “Replay-based CL methods overcome forgetting by saving samples of previous tasks in a replay buffer (Rebuffi et al., 2017; Rolnick et al., 2019; Wang et al., 2021; Yan et al., 2021b) and using them to regularize the learning of new tasks.”
• Deep Class-Incremental Learning: A Survey (2023, background): “Generative Replay [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58]”
• Continually Learning Self-Supervised Representations with Projected Functional Regularization (2021, methodology): “Replaybased methods save a small amount of data from previously seen tasks [4, 9] or generate synthetic data with a generative model [62, 70].”
• Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning (2022, unknown): “…(Castro et al., 2018; He et al., 2018; Javed & Shafait, 2018; Rebuffi et al., 2017; Belouadah & Popescu, 2019; 2020) On the other hand, recent investigations (Wang et al., 2021; He & Zhu, 2021) also explore other alternative possibilities of leveraging unlabelled data to boost continual learning.”
• Class-Incremental Learning: A Survey (2023, background): “Reference § 3.1 Data Replay Direct Replay [26], [33], [34], [35], [36], [37], [38], [39], [40] Generative Replay [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52] § 3.2: Data Regularization [34], [53], [54], [55], [56], [57], [58] § 3.3 Dynamic Networks Neuron Expansion [45],…”
• Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation (2024, methodology): “Replay-based methods [6, 9, 48, 60, 63] exploit an additional memory buffer to save a subset of old training samples, to recover previous data distributions.”