Memory Efficient Experience Replay for Streaming Learning

By Tyler L. Hayes, Nathan D. Cahill, Christopher Kanan. 2018

266 citations recorded by the source corpus.

In supervised machine learning, an agent is typically trained once and then deployed. While this works well for static settings, robots often operate in changing environments and must quickly learn new things from data streams. In this paradigm, known as streaming learning, a learner is trained online, in a single pass, from a data stream that cannot be assumed to be independent and identically distributed (iid). Streaming learning will cause conventional deep neural networks (DNNs) to fail for two reasons: 1) they need multiple passes through the entire dataset; and 2) non-iid data will cause catastrophic forgetting. An old fix to both of these issues is rehearsal. To learn a new example, rehearsal mixes it with previous examples, and then this mixture is used to update the DNN. Full rehearsal is slow and memory intensive because it stores all previously observed examples, and its effectiveness for preventing catastrophic forgetting has not been studied in modern DNNs. Here, we describe the ExStream algorithm for memory efficient rehearsal and compare it to alternatives. We find that full rehearsal can eliminate catastrophic forgetting in a variety of streaming learning settings, with ExStream performing well using far less memory and computation.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Continuous Learning in Single-Incremental-Task Scenarios (2018, methodology): “Figure 1: Venn diagram of some of the most popular CL strategies: CWR [19], PNN [29], EWC [14], SI [33], LWF [17], ICARL [28], GEM [21], FN [11], GDM [26], EXSTREAM [5] and AR1, hereby proposed.”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, neutral): “ER [7, 13], GDumb [45], BiC [62], DER++ [3] and Co(2)L [4].”
Learning to Prompt for Continual Learning (2021, methodology): “We select 5 advanced rehearsal-based methods to compare, including ER [8, 17], GDumb [46], BiC [61], DER++ [3] and Co(2)L [4].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “Hence, many works propose methods for maintaining the buffer [29, 66, 37] or selecting the sam-”
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2021, methodology): “Experience replay with stored data [2, 3, 8, 9, 15, 19, 24, 29, 40, 46, 47, 48, 59] or samples from a generative model [27, 28, 45, 51, 56] is highly effective when storing training data or training/saving a generative model is possible, unlike the DFCIL setting.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…2017; Castro, Marín-Jiménez, Guil, Schmid, & Ala-hari, 2018; Chaudhry et al., 2018; Nguyen, Li, Bui, & Turner, 2018; Hou et al., 2018, 2019; Hayes et al., 2019; Wu, Chen et al., 2019; Lee, Lee, Shin, & Lee, 2019; Belouadah & Popescu, 2019, 2020; Chaudhry et al., 2019; Riemer et al., 2019;…”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “We focus on online continual learning because these systems are more general and existing implementations can learn from data presented in any order [28, 29, 27], while most incremental batch learning implementations are bespoke to incremental class learning and cannot be used for other orderings wi”
Online Continual Learning in Image Classification: An Empirical Survey (2021, methodology): “ER refers to a simple but effective replay-based method that has been discussed in [33, 88].”
Continual Learning in Low-rank Orthogonal Subspaces (2020, background): “Some works [Hayes et al., 2018, Riemer et al., 2019, Rolnick et al., 2018, Chaudhry et al., 2019b, 2020] have shown that directly optimizing the loss on the episodic memory, also known as experience replay, is cheaper than constraint-based approaches and improves prediction performance.”
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2020, methodology): “This simple method, known as experience replay, has been explored and shown to be effective (Rebufﬁ et al., 2017; Wu et al., 2019; Castro et al., 2018; Rajasegaran et al., 2020; Ebrahimi et al., 2020b; Hayes et al., 2019; Riemer et al., 2018).”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “In all comparable metrics of C-F1, O-F1 and mAP, PRS outperforms CRS [72], GSS [3] and even ExStream [35] that uses prior task information to pre-allocate the memory.”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, methodology): “While [26] compare several data selection methods on streaming, to the best of our knowledge, no thorough analysis has been conducted on the effect of the data summarization strategy in replay memory-based continual learning, which we address with an extensive experimental study.”
REMIND Your Neural Network to Prevent Catastrophic Forgetting (2019, methodology): “For iid, the dataset is randomly We compare REMIND to ExStream [14], SLDA [40], an ofﬂine baseline, and a simple baseline where VQA models are ﬁne-tuned without a buffer, which causes catastrophic forgetting.”
Using Hindsight to Anchor Past Knowledge in Continual Learning (2019, methodology): “More recently, several works (Hayes et al., 2018; Riemer et al., 2019; Rolnick et al., 2018; Chaudhry et al., 2019b) have shown that directly optimizing the loss on the episodic memory, also known as experience replay, is cheaper than constraint-based approaches and improves prediction performance.”
Continual Learning with Tiny Episodic Memories (2019, methodology): “More recently, (Hayes et al., 2018; Riemer et al., 2019) used ER for supervised CL learning tasks.”
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis (2019, methodology): “Recently, there have been two notable attempts to marry streaming learning with DNNs: 1) the gradient episodic memory (GEM) family of algorithms [11, 36] and 2) ExStream [23].”