Provable Lifelong Learning of Representations

By Xinyuan Cao, Weiyang Liu, Santosh Srinivas Vempala. 2021

20 citations recorded by the source corpus.

In lifelong learning, tasks (or classes) to be learned arrive sequentially over time in arbitrary order. During training, knowledge from previous tasks can be captured and transferred to subsequent ones to improve sample efficiency. We consider the setting where all target tasks can be represented in the span of a small number of unknown linear or nonlinear features of the input data. We propose a lifelong learning algorithm that maintains and refines the internal feature representation. We prove that for any desired accuracy on all tasks, the dimension of the representation remains close to that of the underlying representation. The resulting sample complexity improves significantly on existing bounds. In the setting of linear features, our algorithm is provably efficient and the sample complexity for input dimension $d$, $m$ tasks with $k$ features up to error $\epsilon$ is $\tilde{O}(dk^{1.5}/\epsilon+km/\epsilon)$. We also prove a matching lower bound for any lifelong learning algorithm that uses a single task learner as a black box. We complement our analysis with an empirical study, including a heuristic lifelong learning algorithm for deep neural networks. Our method performs favorably on challenging realistic image datasets compared to state-of-the-art continual learning methods.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “…and if n s, learning a new task would be more sample-efficient as the dimension of the search space reduces from n to a much smaller s. Li et al. (2022) and Cao et al. (2022) recently explored this idea in the continual learning context with certain new insights, while their settings are from ours.”
Continual Learning by Modeling Intra-Class Variation (2022, methodology): “In addition to the simple network expansion, Rajasegaran et al. (2019); Hung et al. (2019); Cao et al. (2022) perform additional steps to freeze or prune the old network parameters to prevent forgetting, while HAT (Serra et al., 2018) utilizes a hard attention mask to constrain the amount of…”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “The works [33, 25, 1, 4] provide theoretical guarantees on lifelong learning.”