Dark Experience for General Continual Learning: a Strong, Simple Baseline

By Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, Simone Calderara. 2020

1,381 citations recorded by the source corpus.

Neural networks struggle to learn continuously, as they forget the old knowledge catastrophically whenever the data distribution changes over time. Recently, Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through Dark Experience Replay, namely matching the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on top of standard benchmarks, we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. To provide a better understanding, we further introduce MNIST-360, a novel GCL evaluation setting.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dacheng Tao
  2. Bing Liu
  3. Jun Zhu
  4. Yoshua Bengio
  5. Sergey Levine
  6. Mohit Bansal
  7. Jianfeng Gao
  8. Trevor Darrell

In Radiology, Nuclear Medicine and Imaging

  1. Gyuhak Kim
  2. Bin Liu
  3. Bing Liu
  4. Bahram Zonooz
  5. Elahe Arani
  6. Eric Eaton
  7. Kaushik Roy
  8. Zixuan Ke

In Meta & Continual Learning

  1. Bing Liu
  2. Liyuan Wang
  3. Zixuan Ke
  4. Xingxing Zhang
  5. Jun Zhu
  6. Gyuhak Kim
  7. Bin Liu
  8. Hang Su

Citations in the corpus