ELLA: An Efficient Lifelong Learning Algorithm

By Paul Ruvolo, Eric Eaton. 2013

394 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Meta & Continual Learning

Citations in the corpus

Overcoming catastrophic forgetting in neural networks (2016, methodology): “The ELLA algorithm described in [Eaton and Ruvolo, 2013] requires computing and inverting matrices with a dimensionality equal to the number of parameters being optimized, therefore it has been mainly applied to linear and logistic regressions.”
Progressive Neural Networks (2016, methodology): “A3C is trained on CPU using multiple threads and has been shown to converge faster than DQN on GPU.”
Learning without Forgetting (2016, methodology): “[7] describe a method to efficiently add new tasks to a multitask system, co-training all tasks while using only new task data.”
Knowledge Flow: Improve Upon Your Teachers (2019, background): “Other related work includes policy distillation (Rusu et al., 2016a), domain adaptation (Pan & Yang, 2010; Long et al., 2015; Tzeng et al., 2015) or lifelong learning (Chen & Liu, 2016; Thrun, 1998; Mitchell et al., 2015; Ruvolo & Eaton, 2013).”
Lifelong Learning with Dynamically Expandable Networks (2017, neutral): “Eaton & Ruvolo (2013) suggest an online lifelong learning framework (ELLA) that is based on an existing multi-task learning formulation (Kumar & Daume III, 2012) that efﬁciently updates latent parameter bases for a sequence of tasks, by removing dependency to previous tasks for the learning of each…”
Selective Experience Replay for Lifelong Learning (2018, background): “The order in which tasks are encountered does impact learning, and several groups have investigated the effects of ordering (Bengio et al. 2009; Ruvolo and Eaton 2013a; Narvekar et al. 2016).”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, background): “Early techniques under lifelong learning mainly perform knowledge transfer but do not tackle CF (Chen and Liu 2014; Ruvolo and Eaton 2013; Benavides-Prado, Koh, and Riddle 2020).”
Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions (2022, background): “The works [33, 25, 1, 4] provide theoretical guarantees on lifelong learning.”
Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks (2021, methodology): “However, these lifelong learning works mainly use the traditional learning methods such as regression (Ruvolo and Eaton, 2013), naive Bayes (Chen et al.”
Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, background): “LL is now regarded the same as CL, but early LL mainly aimed at improving the new task learning through forward transfer without tackling CF (Silver et al., 2013; Ruvolo and Eaton, 2013; Chen and Liu, 2018).”
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, methodology): “The first part of the definition is taken from (Chen and Liu 2018) based on the early work in (Thrun 1998; Silver, Yang, and Li 2013; Ruvolo and Eaton 2013; Chen and Liu 2014) and the second part of on-the-job learning is proposed in (Liu 2020).”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, methodology): “by Jorge A. Mendez , Boyu Wang , and Eric Eaton”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, background): “A consequence of this format for evaluation is that many continual learning techniques make use of the boundaries between tasks in order to consolidate knowledge during training [Ruvolo and Eaton, 2013, Kirkpatrick et al., 2017, Zenke et al., 2017].”