Continual learning of context-dependent processing in neural networks

By Guanxiong Zeng, Yang Chen, Bo Cui, Shan Yu. 2018

401 citations recorded by the source corpus.

Deep neural networks are powerful tools in learning sophisticated but fixed mapping rules between inputs and outputs, thereby limiting their application in more complex and dynamic situations in which the mapping rules are not kept the same but change according to different contexts. To lift such limits, we developed an approach involving a learning algorithm, called orthogonal weights modification, with the addition of a context-dependent processing module. We demonstrated that with orthogonal weights modification to overcome catastrophic forgetting, and the context-dependent processing module to learn how to reuse a feature representation and a classifier for different contexts, a single network could acquire numerous context-dependent mapping rules in an online and continual manner, with as few as approximately ten samples to learn each. Our approach should enable highly compact systems to gradually learn myriad regularities of the real world and eventually behave appropriately within it. When neural networks are retrained to solve more than one problem, they tend to forget what they have learned earlier. Here, the authors propose orthogonal weights modification, a method to avoid this so-called catastrophic forgetting problem. Capitalizing on such an ability, a new module is introduced to enable the network to continually learn context-dependent processing.

Subfields: Artificial Intelligence, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Alternatively, gradient projection can also be performed without storing old training samples [51], [65], [66], [67].”
Learnability and Algorithm for Continual Learning (2023, methodology): “Orthogonal projection methods learn each task in an orthogonal space to other tasks to reduce task interference or CF (Zeng et al., 2019; Kim & Liu, 2020; Chaudhry et al., 2020; Lin et al., 2022).”
The Ideal Continual Learner: An Agent That Never Forgets (2023, methodology): “For continual linear regression, the methods of Zeng et al. (2019); Saha et al. (2021); Wang et al. (2021b); Kong et al. (2022) are of the form (9), and so they all converge “approximately” to global minimizers of (7); these methods differ mainly in how the projection Kt−1K>t−1 is approximated and…”
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer (2022, methodology): “(1) EWC [12]: a regularization-based method that leverages Fisher Information matrix for weights importance evaluation; (2) HAT [26]: learns a hard attention mask to preserve the knowledge of old task; (3) Orthogonal Weight Modulation (OWM) [32]: learns a projector matrix to project the gradient of ”
Class Gradient Projection For Continual Learning (2022, methodology): “Recently, the gradient projection based paradigms [14, 35, 44] have achieved remarkable performance in tackling the catastrophic forgetting problem in CL.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).”
Balancing Stability and Plasticity through Advanced Null Space in Continual Learning (2022, methodology): “OWM [57] modified the parameters in the direction orthogonal to the input space of previous tasks.”
Memory Efficient Continual Learning with Transformers (2022, methodology): “…methods [27, 2, 23] add a regularization term to the loss to consolidate previous knowledge when learning a new task; (3) Gradient-based methods [68, 1] ensure the gradient updates occur only in the orthogonal direction to the input of old tasks and thus will not affect old tasks.”
TRGP: Trust Region Gradient Projection for Continual Learning (2022, methodology): “As shown in Table 9, for CIFAR-100 Split, TRGP takes around 65% more time than GPM, is comparable with HAT and ER Res, and takes less time than OWM and EWC; for 5-Datasets, TRGP takes around 21% more time than GPM, but is much faster than other baselines including EWC, HAT, A-GEM and ER Res; for Min”
Continual Learning with Recursive Gradient Optimization (2022, methodology): “OWM uses similar iteratively updated projectors derived from recursively least square(RLS), which regards each layer as an independent linear classifier and uses the output of the previous layer to build the projection matrix.”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “OWM [43] is orthogonal weight modiﬁcation method to overcome catastrophic forgetting.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “[43] proposed an Orthogonal Weights Modification (OWM) algorithm.”
Natural continual learning: success is a journey, not (just) a destination (2021, methodology): “…projection matrix P L designed to approximate the posterior covariance of previous tasks Λ − 1 k − 1 ≈ P R ⊗ P L (i.e., the prior covariance on task k ; Appendix D), while Zeng et al. (2019) use the identity matrix I and Duncker et al. (2020) use the covariance of recurrent inputs (Appendix F).”
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, methodology): “HAT, UCL, EWC and OWM were originally designed for image classiﬁcation.”
Training Networks in Null Space of Feature Covariance for Continual Learning (2021, methodology): “Different from GEM and A-GEM, OWM [54] projects the parameter up-date into the orthogonal space of the space spanned by input features of each linear layer.”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, neutral): “Recently, there has been a large body of research in incremental learning [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43].”
GAN Memory with No Forgetting (2020, methodology): “To alleviate catastrophic forgetting, many methods have been proposed, with most focusing on discriminative/classification tasks [36, 65, 95, 55, 94].”
Continual Learning with Knowledge Transfer for Sentiment Classification (2021, methodology): “Orthogonal Weights Modification (OWM) [32] is a state-of-the-art class continual learning (CCL) method.”
CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks (2021, methodology): “HAT, UCL, EWC, OWM and DER++ were originally designed for image classification.”