Learning with Selective Forgetting

By Takashi Shibata, Go Irie, Daiki Ikami, Yu Mitsuzumi. 2021

82 citations recorded by the source corpus.

Lifelong learning aims to train a highly expressive model for a new task while retaining all knowledge for previous tasks. However, many practical scenarios do not always require the system to remember all of the past knowledge. Instead, ethical considerations call for selective and proactive forgetting of undesirable knowledge in order to prevent privacy issues and data leakage. In this paper, we propose a new framework for lifelong learning, called Learning with Selective Forgetting, which is to update a model for the new task with forgetting only the selected classes of the previous tasks while maintaining the rest. The key is to introduce a class-specific synthetic signal called mnemonic code. The codes are "watermarked" on all the training samples of the corresponding classes when the model is updated for a new task. This enables us to forget arbitrary classes later by only using the mnemonic codes without using the original data. Experiments on common benchmark datasets demonstrate the remarkable superiority of the proposed method over several existing methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xinchao Wang
2. Mingli Song
3. Jingwen Ye
4. Jie Song
5. Xingyi Yang
6. Bo Liu
7. Qian Liu
8. René Vidal

In Computer Vision and Pattern Recognition

1. Xinchao Wang
2. Mingli Song
3. Jingwen Ye
4. Jie Song
5. Xingyi Yang
6. Qian Liu
7. Bo Liu
8. Mohan Kankanhalli

In Meta & Continual Learning

1. René Vidal
2. Peter Stone
3. Bo Liu
4. Mohan Kankanhalli
5. Qian Liu
6. Paris V. Giampouras
7. Xinchao Wang
8. Liangzu Peng

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “MnemonicCode[311]embedsa class-specificcodewhenlearningeachclass,enablingtoselec-tivelyforgetthemthroughdiscardingthecorrespondingcodes.”
• The Ideal Continual Learner: An Agent That Never Forgets (2023, positive): “The point here is that, this classic line of research furnished the very rudimentary idea of what is recently known as selective forgetting or active forgetting in the deep continual learning context (Aljundi et al., 2018; Wang et al., 2021a; Shibata et al., 2021; Liu et al., 2022).”
• Learning with Recoverable Forgetting (2022, positive): “Here we tested LIRF on these two tasks, following the setting of LWSF [40], whose task is to unlearn several classes of samples while dealing with the new classes.”
• Safe Distillation Box (2021, background): “The key embedding strategy does not harm the model’s performance, which has been proved in the previous works (Zhang et al. 2018a; Shibata et al. 2021).”
• A Survey of Machine Unlearning (2022, unknown): “) CIFAR 60K 163MB Classification 16+ papers ([58, 70, 136, 163], .”
• Continual Learning and Private Unlearning (2022, background): “In the end, we present the evaluation results by comparing CLPU-DER++ against the following baseline methods: sequential learning (Seq), indepdent learning (Ind), Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), Learning without Forgetting (LwF) (Li & Hoiem, 2017) , Experience Replay (”
• Fast Yet Effective Machine Unlearning (2021, background): “Several other notable works include [39], [40], [41], [42].”