Meta-Learning Requires Meta-Augmentation

By Janarthanan Rajendran, A. Irpan, Eric Jang. 2020

110 citations recorded by the source corpus.

Meta-learning algorithms aim to learn two components: a model that predicts targets for a task, and a base learner that quickly updates that model when given examples from a new task. This additional level of learning can be powerful, but it also creates another potential source for overfitting, since we can now overfit in either the model or the base learner. We describe both of these forms of metalearning overfitting, and demonstrate that they appear experimentally in common meta-learning benchmarks. We then use an information-theoretic framework to discuss meta-augmentation, a way to add randomness that discourages the base learner and model from learning trivial solutions that do not generalize to new tasks. We demonstrate that meta-augmentation produces large complementary benefits to recently proposed meta-regularization techniques.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “Interference: this problem is discussed in (Rolnick et al. 2019) and has been studied in multi-task (for example (Bishop and Svensén 2012; Lin et al. 2019)) and metalearning (for example, (Rajendran, Irpan, and Jang 2020)).”
How to Train Your MAML to Excel in Few-Shot Classification (2021, methodology): “While this randomness has been shown crucial in meta-training to help MAML prevent over-fitting (Rajendran et al., 2020; Yao et al., 2021; Yin et al., 2020), we find that it makes the meta-testing phase unstable.”
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021, background): “Task augmentation for mitigating overﬁtting in meta-learners is ﬁrst studied in Rajendran et al. (2020) in the context of few-shot label adaptation.”