Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis

By Tyler L. Hayes, Christopher Kanan. 2019

175 citations recorded by the source corpus.

When an agent acquires new information, ideally it would immediately be capable of using that information to understand its environment. This is not possible using conventional deep neural networks, which suffer from catastrophic forgetting when they are incrementally updated, with new knowledge overwriting established representations. A variety of approaches have been developed that attempt to mitigate catastrophic forgetting in the incremental batch learning scenario, where a model learns from a series of large collections of labeled samples. However, in this setting, inference is only possible after a batch has been accumulated, which prohibits many applications. An alternative paradigm is online learning in a single pass through the training dataset on a resource constrained budget, which is known as streaming learning. Streaming learning has been much less studied in the deep learning community. In streaming learning, an agent learns instances one-by-one and can be tested at any time, rather than only after learning a large batch. Here, we revisit streaming linear discriminant analysis, which has been widely used in the data mining research community. By combining streaming linear discriminant analysis with deep learning, we are able to outperform both incremental batch learning and streaming learning algorithms on both Ima- geNet ILSVRC-2012 and CORe50, a dataset that involves learning to classify from temporally ordered samples1.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

Learnability and Algorithm for Continual Learning (2023, neutral): “The exemplarfree methods are: HAT (Serrà et al., 2018), OWM (Zeng 2https://github.com/k-gyuhak/CLOOD et al., 2019), SLDA (Hayes & Kanan, 2020), PASS (Zhu et al., 2021), and L2P (Wang et al., 2022c).”
First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, background): “We show that surprisingly few samples in the first session are sufficient for adaptation and that our solution is more powerful than current few-shot continual learning solutions.”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, methodology): “Please note that our CA or CA+LN is operated in a post-hoc fashion rather than aligning the classifier during representation learning [55, 11, 12], as the latter would even worsen the performance ( e.g ., by 27.89% on Split CIFAR-100).”
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning (2022, background): “Other works have looked at rehearsal-free continual learning from an online “streaming” learning perspective using a frozen, pre-trained model [20, 40].”
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, methodology): “Several methods in CL use pretrained backbones as feature extractors such as in Hayes et al [7] or [1,9] and some-times the pretraining is carried on half (or a big portion) the dataset considered, as in PODNet [4] or in Yu et al. [26].”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Evaluation Metrics: Following prior works, we evaluate methods in the class-incremental learning setting using: (I) final performance, or the performance with respect to all past classes after having seen all N tasks (referred to as AN,1:N ); and (II) Ω, or the average (over all tasks) normalized ta”
Self-Supervised Training Enhances Online Continual Learning (2021, methodology): “We study the common continual learning paradigm in which pre-training precedes continual learning [28, 29, 55, 10, 71, 32, 33, 34, 68, 4].”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “DSLDA [124] freezes the feature extractor and uses deep Streaming Linear Discriminant Analysis [125] to train the output layer incrementally.”
EEC: Learning to Encode and Regenerate Images for Continual Learning (2021, methodology): “To avoid storing old class images, some approaches store features from the last fully-connected layer of the neural networks (Xiang et al., 2019; Hayes & Kanan, 2020; Ayub & Wagner, 2020b;d).”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “We compare REMIND to ExStream [22], SLDA [23], an offline baseline, and a simple baseline where models are fine-tuned without a buffer, which causes catastrophic forgetting.”
Latent Replay for Real-Time Continual Learning (2019, methodology): “Deep Streaming Linear Discriminant Analysis (DSLDA) was recently proposed [23] where an online extension of the LDA classifier works on the top of a fixed deep learning feature extractor.”
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, positive): “We evaluate our methods using: (I) ﬁnal performance, or the performance with respect to all past classes after having seen all N tasks (referred to as A N, 1: N ); and (II) Ω , or the average (over all tasks) normalized task accuracy with respect to an ofﬂine oracle method [23].”
Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches (2019, methodology): “• we run several experiments to evaluate the proposed strategies (CWR* and AR1*) and to compare them with two baselines and three state-of-the-art rehearsalfree approaches (such as EWC [14], LWF [16] and DSLDA [6]), also in terms of computation and memory efficiency.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, methodology): “L without memory [56]. We note that these methods work better than distillation based IL algorithms even when initial representations are learned with a few dozens of classes. Online learning methods [23,24] are useful when the stream of data arrives image per image. The model capacity to learn indi”
RODEO: Replay for Online Object Detection (2020, neutral): “While streaming learning has recently been used for image classification [7, 15, 16, 17, 36], it has not yet been explored for object detection, which we pioneer here.”
A Multi-Head Model for Continual Learning via Out-of-Distribution Replay (2022, methodology): “There are many existing online CL systems, e.g., SLDA (Hayes & Kanan, 2020), REMIND (Hayes et al., 2020) and OCM (Guo et al., 2022).”
Online Continual Learning for Embedded Devices (2022, methodology): “This architecture was chosen since it is a common architecture used in continual learning literature (Rebuffi et al., 2017; Castro et al., 2018; Hayes et al., 2020; Douillard et al., 2020; Wu et al., 2019).”
Do We Need Fully Connected Output Layers in Convolutional Networks? (2020, background): “For supervised lifelong machine learning applications [6, 7, 14, 24], the number of categories increases over time so having the number of parameters increase sub-linearly with the number of classes learned is especially important.”
CGLB: Benchmark Tasks for Continual Graph Learning (2022, background): “In the past, continual learning has been explored in many areas including computer vision [19, 51], reinforcement learning [38], etc. Essentially, it targets at resolving the catastrophic forgetting problem, i.e. the drastic decrease of a model’s performance on previous tasks after being trained on…”