Memory Efficient Meta-Learning with Large Images

By John F. Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, Richard E. Turner. 2021

21 citations recorded by the source corpus.

Meta learning approaches to few-shot classification are computationally efficient at test time, requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task's training images. This enables us to perform a forward pass on a task's entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test-time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning (2023, positive): “In addition to the results presented here, we have also tested meta-learning based body adaptation methods [3, 26, 2] which support continual learning out of the box.”
Meta-learning without data via Wasserstein distributionally-robust model fusion (2022, background): “…Rajeswaran et al., 2019, Finn et al., 2019b, Raghu et al., 2020, Wang et al., 2020b, Zhou et al., 2021, Bohdal et al., 2021, Rothfuss et al., 2021, CHEN et al., 2021, Bronskill et al., 2021, Sun et al., 2021] focus on the ofﬂine setting, where all the training tasks are available together upfront.”
FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, methodology): “The results for all models (FineTuner (Yosinski et al., 2014), Simple CNAPs (Bateni et al., 2020) and ProtoNets (Snell et al., 2017)) are from (Bronskill et al., 2021).”