New Metrics and Experimental Paradigms for Continual Learning

By Tyler L. Hayes, Ronald Kemker, Nathan D. Cahill, Christopher Kanan. 2018

37 citations recorded by the source corpus.

In order for a robotic agent to learn successfully in an uncontrolled environment, it must be able to immediately alter its behavior. Deep neural networks are the dominant approach for classification tasks in computer vision, but typical algorithms and architectures are incapable of immediately learning new tasks without catastrophically forgetting previously acquired knowledge. There has been renewed interest in solving this problem, but there are limitations to existing solutions, including poor performance compared to offline models, large memory footprints, and learning slowly. In this Abstract, we formalize the continual learning paradigm and propose new benchmarks for assessing continual learning agents.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Meta & Continual Learning

Citations in the corpus

Continual Learning with Fully Probabilistic Models (2021, background): “The ﬁeld of CL is expanding rapidly, see [1, 2, 3, 4] for reviews.”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “d from Di as in [32, 33], where concept drift may occur within Di.”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “es across all orderings. Since CORe50 is small, CNNs are rst initialized with pre-trained ImageNet weights and then ne-tuned on a subset of 1200 samples for base initialization. We use the all metric [22,24,45] for evaluation, which normalizes incremental learning performance by oine performance: al”
Continual Learning for Robotics (2019, background): “While forgetting and knowledge transfer could be quantiﬁed and evaluated in various ways, as argued in [41, 56, 71], these may not suﬃce for a robust evaluation of CL strategies.”
Don't forget, there is more than forgetting: new metrics for Continual Learning (2018, background): “An overall performance Ω in a supervised classiﬁcation setting was proposed in (Hayes et al., 2018) based on the relative performance of an incrementally trained algorithm with respect to an off-line trained one (which has access to all the data at once, and acts as an upper bound).”
CVPR 2020 Continual Learning in Computer Vision Competition: Approaches, Results, Current Challenges and Future Directions (2020, background): “Moreover, a number of researchers already pointed out how different training/testing regimes impacts forgetting and the continual learning process (Mirzadeh et al., 2020; Maltoni & Lomonaco, 2019; Hayes et al., 2018).”
Online Continual Learning on Sequences (2020, background): “h training schemes and performance metrics are better to evaluate CL models. Diﬀerent sets of metrics have been proposed to evaluate CL performance on supervised and unsupervised learning tasks (e.g. [21,28,7]). In the absence of standardized metrics and evaluation schemes, it is unclear what it mea”
Partitioned Variational Inference: A unified framework encompassing federated and continual learning (2018, background): “…et al., 2013, Li and Hoiem, 2016, Kirkpatrick et al., 2017, Zenke et al., 2017, Seff et al., 2017, Bui et al., 2017a, Nguyen et al., 2018, Zeno et al., 2018, Chaudhry et al., 2018], to setting up relevant metrics and benchmarks for evaluation [Lomonaco and Maltoni, 2017, Hayes et al., 2018].”