On Anytime Learning at Macroscale

By Lucas Caccia, Jing Xu, Myle Ott, Marc'Aurelio Ranzato, Ludovic Denoyer. 2021

29 citations recorded by the source corpus.

In many practical applications of machine learning data arrives sequentially over time in large chunks. Practitioners have then to decide how to allocate their computational budget in order to obtain the best performance at any point in time. Online learning theory for convex optimization suggests that the best strategy is to use data as soon as it arrives. However, this might not be the best strategy when using deep non-linear networks, particularly when these perform multiple passes over each chunk of data rendering the overall distribution non i.i.d.. In this paper, we formalize this learning setting in the simplest scenario in which each data chunk is drawn from the same underlying distribution, and make a first attempt at empirically answering the following questions: How long should the learner wait before training on the newly arrived chunks? What architecture should the learner adopt? Should the learner increase capacity over time as more data is observed? We probe this learning setting using convolutional neural networks trained on classic computer vision benchmarks as well as a large transformer model trained on a large-scale language modeling task. Code is available at \url{www.github.com/facebookresearch/ALMA}.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Unified Scaling Laws for Routed Language Models (2022, neutral): “In studying the empirical aspects of scaling, this work follows Kaplan et al. [2020]; which triggered much research including Henighan et al.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “In a setting where training data is streamed, Caccia et al. (2021) deﬁne a growing ensemble , in which new base models are trained sequentially on incoming batches.”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, background): “A similar observation has been made recently by Caccia et al. (2021) in their “anytime learning“ framework where data arrive by batches instead of a whole dataset.”
Scaling the Number of Tasks in Continual Learning (2022, unknown): “Our evaluation protocol is also similar to ALMA [Caccia et al., 2021] scenario, since we evaluate on a ﬁxed test set.”