Online Incremental Feature Learning with Denoising Autoencoders

By Guanyu Zhou, Kihyuk Sohn, Honglak Lee. 2012

223 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Lifelong Learning with Dynamically Expandable Networks (2017, neutral): “There are multiple ways to perform such incremental learning (Rusu et al., 2016; Zhou et al., 2012).”
Policy Consolidation for Continual Reinforcement Learning (2019, background): “…the amount of neural resources, some utilise the deﬁnition of task boundaries (Rusu et al., 2016; Xu & Zhu, 2018), while others do not (Ring, 1997; Zhou et al., 2012), but since the computational complexity grows with the size of the network, it is not clear that they are scalable for continual…”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “These method can be roughly divided into four categories: architecture-based methods expand the network or allocate new neurons for new tasks [13,28,42,60]; replayedbased methods interleave old data with current data by storing historical data in a buffer or generating virtual old data [4,8,12,39,40”
Continual Unsupervised Representation Learning (2019, background): “Expansion-based models dynamically increase capacity to allow for additional tasks (Rusu et al., 2016; Yoon et al., 2017; Draelos et al., 2017), and optionally prune the network to constrain capacity (Zhou et al., 2012; Golkar et al., 2019).”
Scalable and Order-robust Continual Learning with Additive Parameter Decomposition (2019, methodology): “Dynamic network expansion approaches have been introduced in earlier work such as Zhou et al. (2012), which proposed an iterative algorithm to train a denoising autoencoder while adding in new neurons one by one and merging similar units.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “The former by adding units whenever the reconstruction error of an autoencoder surpasses a predetermined threshold in the spirit of Zhou et al. (2012), the latter based on an empirically found value of the classiﬁcation loss in supervised learning.”
Progressive Neural Networks (2016, background): “This made it a more natural ﬁt for the large amount of sequential experiments required for this work.”
Online Deep Learning: Growing RBM on the fly (2018, methodology): “For example, online algorithms for denoising autoencoders (DAE) [20] have been used for incremental feature learning with streaming data, but need a priori training with a DAE architecture as the building block to learn a base set of features first.”
Deep learning applications and challenges in big data analytics (2015, methodology): “In this section, we discuss some works associated with Deep Learning and streaming data, including incremental feature learning and extraction [49], denoising autoencoders [50], and deep belief networks [51].”
Continual Adaptation of Visual Representations via Domain Randomization and Meta-learning (2020, background): “…approaches into three categories: (i) dynamic architectures, where the model’s underlying architecture is modiﬁed as it learns new patterns [52, 65, 10, 60, 15, 62, 48]; (ii) rehearsal methods, that rely on memory replay and overcome catastrophic forgetting by storing samples from old…”
Learning the Structure of Deep Convolutional Networks (2015, methodology): “[29] also developed a greedy method to determine the suitable number of filters for denoising auto-encoders.”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, methodology): “First, Zhou et al. [236] employed the denoising Auto-Encoders for online learning, which is capable of learning new patterns at the cost of slow learning speed due to large parameters space and a lack of support for parallel execution.”
Adaptive Deep Models for Incremental Learning: Considering Capacity Scalability and Sustainability (2019, background): “proposed an incremental feature learning algorithm to determine the optimal model complexity based on the autoencoder [38].”
Online Deep Learning: Learning Deep Neural Networks on the Fly (2017, background): “There have been attempts at making deep learning compatible with online learning (Zhou, Sohn, and Lee 2012; Lee et al. 2016) and (Lee et al. 2017).”