Unsupervised Progressive Learning and the STAM Architecture

By James Smith, Cameron Taylor, Seth Baer, C. Dovrolis. 2019

41 citations recorded by the source corpus.

We first pose the Unsupervised Progressive Learning (UPL) problem: an online representation learning problem in which the learner observes a non-stationary and unlabeled data stream, learning a growing number of features that persist over time even though the data is not stored or replayed. To solve the UPL problem we propose the Self-Taught Associative Memory (STAM) architecture. Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical features rather than specific examples. We evaluate STAM representations using clustering and classification tasks. While there are no existing learning scenarios that are directly comparable to UPL, we compare the STAM architecture with two recent continual learning models, Memory Aware Synapses (MAS) and Gradient Episodic Memories (GEM), after adapting them in the UPL setting.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

A soft nearest-neighbor framework for continual semi-supervised learning (2022, neutral): “While the vast majority of these methods operate in a supervised setting, recent works addressed the problem of overcoming catastrophic forgetting in the challenging case of limited or no supervision [1, 8, 21, 28, 38, 46, 47, 53].”
Self-Supervised Models are Continual Learners (2021, background): “Works such as [1, 42, 48] laid the foundations of unsupervised CL, but their studies are severely limited to digitlike datasets, e.”
Representational Continuity for Unsupervised Continual Learning (2021, background): “Rao et al. (2019); Smith et al. (2021) tackled the problem of continual unsupervised representation learning; however, their methods are restricted to simple low-resolution tasks and not scalable to large-scale continual learning datasets.”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “For example, methods which expand a model’s architecture as new tasks are encountered are highly effective for applications where a model growing with tasks is practical [12, 32, 36, 40, 48, 51].”
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, positive): “Architectural approaches typically avoid over-writing the current model by expanding the model parameters to make room for knowledge related to novel tasks [44, 45, 46, 47, 48, 49].”
Unsupervised Class-Incremental Learning Through Confusion (2021, background): “Smith & Dovrolis (2019) performs novelty detection by analyzing an input image through a series of receptive ﬁelds to determine if an input patch is an outlier.”
SPeCiaL: Self-Supervised Pretraining for Continual Learning (2021, background): “Unsupervised Continual Learning Several works [ Rao et al. , 2019; Smith et al. , 2019; Achille et al. , 2018 ] learn representations in an unsupervised manner over non-iid data.”
Unsupervised Continual Learning Via Pseudo Labels (2021, background): “In addition, CURL [38] and STAM [45] are proposed to address unsupervised continual learning under task-free setting [3] where task boundary is not given.”
Self-Supervised Learning Aided Class-Incremental Lifelong Learning (2020, background): “Catastrophic forgetting firstly explored in 1980s and 1990s [7, 29, 37], has attracted more and more attentions in recent years and lifelong learning has also been extended to unsupervised settings [35] and semi-supervised settings [42].”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, methodology): “Our methods operate over handcrafted representations of the observation space; the system could be extended by introducing representation learning from RGB-observations via unsupervised (STAM (Smith et al., 2019a;b)) and self-supervised learning (Gallardo et al., 2021).”