PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

By Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, Eduardo Valle. 2020

909 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “matic representation shifts, PODNet [100] employs a spatial distillation loss to preserve representations throughout the model.”
S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learning (2023, methodology): “Several state-of-art-works [38,45,2,12,3] proposed different techniques to address the class imbalance problem in CIL like rescaling scores or balanced finetuning of classifiers, etc.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, methodology): “We follow the benchmark protocol used in [13, 18, 30, 31].”
A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “The first category comprises regularization-based methods [13,14,20,22,24,25,30,45,54], which address the problem of catastrophic forgetting by introducing appropriate regularization terms in the objective function or identifying a set of parameters that are most relevant for certain tasks.”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, methodology): “For ImageNet-Subset, FeTrIL accuracy is better than LUCIR’s, slightly behind that of Mnemonics [22] and approximately 3.5 points lower than that of PODNet [7].”
Generative Negative Text Replay for Continual Vision-Language Pretraining (2022, methodology): “In addition, the feature-based distillation methods [15] are usually designed for uni-modal CNNs and rarely take into account the characteristics of transformer-based multi-modal pre-training, and the model inversion [51] aims to generate images as positive examples for certain classes using a froze”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “On the contrary, class-incremental methods [4, 8, 33, 36] attempt to recognize all categories observed so far at once without the task identiﬁer, being more practical.”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “A number of methods (Wu et al., 2019; Hou et al., 2019; Liu et al., 2020; Douillard et al., 2020) take a path similar to iCaRL and utilize an extra model with a memory buffer to prevent forgetting.”
Long-Tailed Class Incremental Learning (2022, methodology): “In this section we analyze the performance of four popular CIL methods: LwF [20] with exempalrs, EEIL [7], LUCIR [13], and PODNET [10]).”
Anti-Retroactive Interference for Lifelong Learning (2022, neutral): “UCIR GEM PODNet[6] iTAML iTAML+RRR ARI”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, methodology): “Specifically, on CIFAR100, adding CSCCT to iCARL [42], LUCIR [19] and PODNet [13] provides strong relative improvement of 2.76%, 2.28% and 1.99% respectively averaged across all settings, while on the much more highdimensional ImageNet-Subset, adding our method to the respective baselines provides c”
Online Continual Learning with Contrastive Vision Transformer (2022, neutral): “Rehearsal-based methods [8,37,81,60,15,22,49,79,68,52] alleviate catastrophic forgetting by replaying a subset of data of past tasks stored in limited buffer.”
Novel Class Discovery without Forgetting (2022, neutral): “Regularization based methods [31,41,9,53,13,34] ensure that the”
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, methodology): “Several methods in CL use pretrained backbones as feature extractors such as in Hayes et al [7] or [1,9] and some-times the pretraining is carried on half (or a big portion) the dataset considered, as in PODNet [4] or in Yu et al. [26].”
Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data (2022, methodology): “For ImageNetSubset, we use the original RseNet-18 as our backbone following prior works [6,12,18,28,32,33].”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, neutral): “PODNet [8]: PODNet proposes a novel spatial-based distillation loss that can be applied throughout the model.”
General Incremental Learning with Domain-aware Categorical Representations (2022, methodology): “The second are the distillation-based methods [4, 9, 13, 28, 30, 35], which adopt knowledge distillation to preserve the representation by penalizing the difference between outputs of previous and current models.”
Class-Incremental Learning with Strong Pre-trained Models (2022, methodology): “Class-incremental learning (CIL) methods [5,6,8,11,23, 29,34] learn to categorize more and more classes over time.”
Energy-based Latent Aligner for Incremental Learning (2022, background): “One popular approach to achieve this behaviour is by constraining the parameters to not deviate much from previously tuned values [7,10,28,32,41,52].”