Don't forget, there is more than forgetting: new metrics for Continual Learning

By Natalia Díaz Rodríguez, Vincenzo Lomonaco, David Filliat, Davide Maltoni. 2018

205 citations recorded by the source corpus.

Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and forward knowledge transfer, memory overhead as well as computational efficiency. Drawing inspiration from the standard Multi-Attribute Value Theory (MAVT) we further propose to fuse these metrics into a single score for ranking purposes and we evaluate our proposal with five continual learning strategies on the iCIFAR-100 continual learning benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

AFEC: Active Forgetting of Negative Transfer in Continual Learning (2021, neutral): “Several studies suggested that forward knowledge transfer is critical for continual learning [17, 4], which might be either positive or negative due to the dynamic data distributions.”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “This approach also signiﬁcantly improves other key metrics, e.g., forward-backward transfer and computational complexity of training and inference that have received limited attention (Díaz-Rodríguez et al., 2018).”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “[42] Natalia Díaz Rodríguez, Vincenzo Lomonaco, David Filliat, and Davide Maltoni.”
Bayesian Structure Adaptation for Continual Learning (2019, background): “We also report results on some additional CL metrics (D´ıaz-Rodr´ıguez et al., 2018) in the Supplementary Material.”
Continual Learning with Adaptive Weights (CLAW) (2019, background): “Indeed, catastrophic forgetting is one of the main pathologies in continual learning (McCloskey and Cohen, 1989; Ratcliff, 1990; Robins, 1993, 1995; French, 1999; Pape et al., 2011; Goodfellow et al., 2014a; Achille et al., 2018; Kemker et al., 2018; Kemker and Kanan, 2018; Diaz-Rodriguez et al., 20”
Towards Robust Evaluations of Continual Learning (2018, background): “Others have worked towards richer metrics for measuring continual learning performance like forward/backwards transfer or learning curve area (Lopezpaz and Ranzato, 2017; Chaudhry et al., 2018; Schwarz et al., 2018b; Diaz-Rodriguez et al., 2018; Chaudhry et al., 2019).”
Continual Learning for Robotics (2019, background): “The metrics are extended for more fine grained, generic evaluation (Díaz-Rodríguez et al., 2018) so that the original accuracy (Lopez-Paz and Ranzato, 2017) (as well as BWT and FWT) can account for performance at every timestep in time.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “made to emphasize the need for more diverse evaluation [6], [13].”
Avalanche: an End-to-End Library for Continual Learning (2021, background): “Avalanche’s effort to monitor different facets of performance aims at enabling a wider experimental assessment, which is too often focused only on the forgetting of previous knowledge [11].”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, background): “…on earlier task Tj , where i > j: Zi,j = ri,j,end − ri,j−1,end |ri,all,max| (5) When Zi,j > 0, the agent has become better at later task Ti having trained on earlier task Tj , indicating forward transfer has occurred by zero-shot learning (Lopez-Paz & Ranzato, 2017; Díaz-Rodríguez et al., 2018).”
Explainability in Deep Reinforcement Learning (2020, background): “This kind of methods optimizing for compositionality does not provide a direct explanation tool; however compositionality can be qualitatively observed as self organized modules [94] and used to train multiple policies that beneﬁt from being combined.”
Challenges in Task Incremental Learning for Assistive Robotics (2020, background): “Overall accuracy is the average value of all elements in A i , j [22].”
Selective Replay Enhances Learning in Online Continual Analogical Reasoning (2021, methodology): “We also adopt three metrics from [19] to evaluate average accuracy and backward/forward transfer.”
Practical Recommendations for Replay-based Continual Learning Methods (2022, background): “Moreover it is diﬃcult to compare these approaches since there is not a standard evaluation protocol [8].”