CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability

By Martin Mundt, Steven Braun, Quentin Delfosse, Kristian Kersting. 2021

42 citations recorded by the source corpus.

What is the state of the art in continual machine learning? Although a natural question for predominant static benchmarks, the notion to train systems in a lifelong manner entails a plethora of additional challenges with respect to set-up and evaluation. The latter have recently sparked a growing amount of critiques on prominent algorithm-centric perspectives and evaluation protocols being too narrow, resulting in several attempts at constructing guidelines in favor of specific desiderata or arguing against the validity of prevalent assumptions. In this work, we depart from this mindset and argue that the goal of a precise formulation of desiderata is an ill-posed one, as diverse applications may always warrant distinct scenarios. Instead, we introduce the Continual Learning EValuation Assessment Compass: the CLEVA-Compass. The compass provides the visual means to both identify how approaches are practically reported and how works can simultaneously be contextualized in the broader literature landscape. In addition to promoting compact specification in the spirit of recent replication trends, it thus provides an intuitive chart to understand the priorities of individual systems, where they resemble each other, and what elements are missing towards a fair comparison.

Subfields: Artificial Intelligence, Health Informatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

Citations in the corpus

Few-Shot Continual Active Learning by a Robot (2022, background): “However, as mentioned in [17, 34], one significant limitation of all CL approaches is that they require the complete training data of each class to be labeled and available in a single increment, however in real-world robotic applications data is available in a streaming manner (online learning [19]”
Continual Learning via Local Module Composition (2021, background): “The rapid growth of continual learning has lead researchers to work on empirical studies [20, 58, 56], surveys [32, 42, 55, 66, 67] as well as CL-specific software [72, 22, 59].”
Three types of incremental learning (2022, unknown): “Because of an abundance of subtle, but often important, differences between evaluation protocols, systematic comparison between continual learning algorithms is challenging, even when papers use the same datasets 9 .”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “In a similar spirit to this survey, Mundt et al. (2022) presented the CLEVA-Compass, a visual representation for analyzing lifelong learning approaches according to a variety of axes.”
Pretrained Language Model in Continual Learning: A Comparative Study (2022, background): “…researchers to work on empirical studies (De Lange et al., 2019; Lesort et al., 2021b), surveys (Hadsell et al., 2020; Khetarpal et al., 2020; Lesort et al., 2021a; Mundt et al., 2020; 2021) as well as CL-speciﬁc software (Normandin et al., 2021; Douillard & Lesort, 2021; Lomonaco et al., 2021).”