Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions

By Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Dongling Zhan, Tiehang Duan, Mingchen Gao. 2022

22 citations recorded by the source corpus.

The paradigm of machine intelligence moves from purely supervised learning to a more practical scenario when many loosely related unlabeled data are available and labeled data is scarce. Most existing algorithms assume that the underlying task distribution is stationary. Here we consider a more realistic and challenging setting in that task distributions evolve over time. We name this problem as Semi-supervised meta-learning with Evolving Task diStributions, abbreviated as SETS. Two key challenges arise in this more realistic setting: (i) how to use unlabeled data in the presence of a large amount of unlabeled out-of-distribution (OOD) data; and (ii) how to prevent catastrophic forgetting on previously learned task distributions due to the task distribution shift. We propose an OOD Robust and knowleDge presErved semi-supeRvised meta-learning approach (ORDER), to tackle these two major challenges. Specifically, our ORDER introduces a novel mutual information regularization to robustify the model with unlabeled OOD data and adopts an optimal transport regularization to remember previously learned knowledge in feature space. In addition, we test our method on a very challenging dataset: SETS on large-scale non-stationary semi-supervised task distributions consisting of (at least) 72K tasks. With extensive experiments, we demonstrate the proposed ORDER alleviates forgetting on evolving task distributions and is more robust to OOD data than related strong baselines.

Subfields: Artificial Intelligence, Cancer Research, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Hang Su
  2. Jun Zhu
  3. Liyuan Wang
  4. Xingxing Zhang
  5. Tianyu Pang
  6. Yinpeng Dong
  7. Vamsi Krishna Ithapu
  8. Mirella Lapata

In Cancer Research

  1. Jun Zhu

In Meta & Continual Learning

  1. Xingxing Zhang
  2. Liyuan Wang
  3. Hang Su
  4. Jun Zhu
  5. Simone Scardapane
  6. Valerio Marsocci
  7. Liyuan Wang
  8. Hongwei Yan

Citations in the corpus