Overcoming Catastrophic Forgetting with Hard Attention to the Task

2018

Subfields: Model Architectures, Meta & Continual Learning

Citations in the corpus

• Meta-attention for ViT-backed Continual Learning (2022, methodology): “LwF, Piggyback, HAT and Adaptor-Bert are four representative existing methods from both computer vision and NLP for continual learing.”
• Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, methodology): “HAT (Serrà et al., 2018) takes the same inputs and use hard attention to learn binary masks to protect old models in the TCL setting.”
• Generalized Variational Continual Learning (2020, methodology): “The implementation of all baseline methods was based on the Github repository 5 for HAT (Serra et al., 2018), except the implementions of IMM-Mode and EWC were modified due to an error in the computation of the Fisher Information Matrix in the original implementation.”
• Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, methodology): “A similar intuition was pointed out by(Aljundi, Rohrbach, and Tuytelaars 2018; Lee et al. 2017a; Serrà et al. 2018; Siavash Golkar 2019).”
• Don't forget, there is more than forgetting: new metrics for Continual Learning (2018, background): “Serrà et al. (2018), instead, try to directly model forgetting with a proposed forgetting ratio metric ρ that considers a vector of test accuracies for each Te i at random initialization that represents a lower bound on accuracy.”
• Online Fast Adaptation and Knowledge Accumulation (OSAKA): a New Approach to Continual Learning (2020, background): “This is done either by freezing the old network [83, 69] or growing new parts of the network [41, 68].”
• SPACE: Structured Compression and Sharing of Representational Space for Continual Learning (2020, methodology): “The concept of successively pruning and/or freezing parameters in a continual learning setup has been explored in many previous works such as in [23], PackNet [20] and HAT [25].”
• CLNeRF: Continual Learning Meets NeRF (2023, unknown): “Parameter isolation methods prevent forgetting by freezing (a subset of) neurons from previous tasks and use new neurons to learn later tasks [19, 29].”
• Not Just Selection, but Exploration: Online Class-Incremental Continual Learning via Dual View Consistency (2022, methodology): “Specifically, Parameter-isolation methods can be divided into two types: Fixed Architecture (FA) [19,37,44] and Dynamic Architecture (DA) [3, 43, 54].”