Computationally Budgeted Continual Learning: What Does Matter?

By Ameya Prabhu, Hasan Hammoud, Puneet Kumar Dokania, Philip H. S. Torr, Ser-Nam Lim, Bernard Ghanem, Adel Bibi. 2023

90 citations recorded by the source corpus.

Continual Learning (CL) aims to sequentially train models on streams of incoming data that vary in distribution by preserving previous knowledge while adapting to new data. Current CL literature focuses on restricted access to previously seen data, while imposing no constraints on the computational budget for training. This is unreasonable for applications in-the-wild, where systems are primarily constrained by computational and time budgets, not storage. We revisit this problem with a large-scale benchmark and analyze the performance of traditional CL approaches in a compute-constrained setting, where effective memory samples used in training can be implicitly restricted as a consequence of limited computation. We conduct experiments evaluating various CL sampling strategies, distillation losses, and partial fine-tuning on two large-scale datasets, namely ImageNet2K and Continual Google Landmarks V2 in data incremental, class incremental, and time incremental settings. Through extensive experiments amounting to a total of over 1500 GPU-hours, we find that, under compute-constrained setting, traditional CL approaches, with no exception, fail to outperform a simple minimal baseline that samples uniformly from memory. Our conclusions are consistent in a different number of stream time steps, e.g., 20 to 200, and under several computational budgets. This suggests that most existing CL methods are particularly too computationally expensive for realistic budgeted deployment. Code for this project is available at: https://github.com/drimpossible/BudgetCL.

Subfields: Artificial Intelligence, Civil and Structural Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

Citations in the corpus

Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “A memory based method is employed to approximate the loss of the previous tasks.”
Class-Incremental Learning: A Survey (2023, background): “Another important characteristic is the computational budget [228].”
Online Continual Learning For Interactive Instruction Following Agents (2024, background): “…recent approaches (Li & Hoiem, 2017; Koh et al., 2023; Boschini et al., 2022b) use knowledge distillation with the trained model until past tasks, but this often entails significant memory (Zhou et al., 2022a) and computational overhead caused by additional model inference (Prabhu et al., 2023).”
Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation (2024, methodology): “Recent work (Prabhu et al., 2023a) has demonstrated the effectiveness of offline continual learning under limited budgets, showing that learning from a balanced distribution is helpful when budgets are insufficient.”
TiC-CLIP: Continual Training of CLIP Models (2023, methodology): “We ablate on T I C-DataComp (M) to investigate how to change LR after training on data from the first time step.”