Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learning

By Joel Joseph, Alex Gu. 2021

77 citations recorded by the source corpus.

The Continual Learning (CL) problem involves performing well on a sequence of tasks under limited compute. Current algorithms in the domain are either slow, offline or sensitive to hyper-parameters. La-MAML, an optimization-based meta-learning algorithm claims to be better than other replay-based, prior-based and meta-learning based approaches. According to the MER paper [1], metrics to measure performance in the continual learning arena are Retained Accuracy (RA) and Backward Transfer-Interference (BTI). La-MAML claims to perform better in these values when compared to the SOTA in the domain. This is the main claim of the paper, which we shall be verifying in this report.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “Withthehelpofexperiencereplay,La-MAML[185] optimizestheOML[181]objectiveinanonlinefashionwith anadaptivelymodulatedlearningrate.”
An Empirical Investigation of the Role of Pre-training in Lifelong Learning (2021, background): “Meta-learning involves the development of models capable of learning over time and has been employed in various studies focusing on lifelong learning (Riemer et al., 2019; Finn et al., 2019; Javed and White, 2019; Wang et al., 2020; Gupta et al., 2020).”
Co2L: Contrastive Continual Learning (2021, background): “Recent approaches [22, 16] leverage meta-learning [14] separate concerns of representation learning and classi-ﬁer training and directly optimizes objectives that minimize forgetting by learning representations that accelerate future learning on meta-learning frameworks.”
Learngene: From Open-World to Your Learning Task (2021, background): “Although meta continual learning (Yao et al. 2020; Gidaris and Komodakis 2018; Jerfel et al. 2019; Javed and White 2019; Gupta, Yadav, and Paull 2020) can solve the sequential task problem by recording the efﬁciency task buffer, it still cannot prevent data-interaction.”
Forget-free Continual Learning with Winning Subnetworks (2022, methodology): “Also, † and ∗ denotes results reported from (Deng et al., 2021) and (Gupta et al., 2020) (b), the capacity of Conv4 is greater than that of Conv1, while the Conv4 proportion of reused weights for all tasks is smaller than Conv1.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “At present, MAML has been widely used in various tasks [101], [102], [103], producing different variants.”
SPeCiaL: Self-Supervised Pretraining for Continual Learning (2021, methodology): “Some replay based methods further aim at directly optimizing for the reduction of negative interference across tasks [ Lopez-Paz and Ranzato, 2017; Riemer et al. , 2018; Chaudhry et al. , 2018; Gupta et al. , 2020 ] .”
Mitigating Catastrophic Forgetting in Online Continual Learning by Modeling Previous Task Interrelations via Pareto Optimization (2024, methodology): “…algorithms have been proposed in continual learning (CL) area, aiming to enhance the incremental learning ability of DNNs on streaming tasks (Lopez-Paz & Ranzato, 2017; Serra et al., 2018; Gupta et al., 2020; Guo et al., 2020; Arani et al., 2022; Wang et al., 2023; Chrysakis & Moens, 2023).”