Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data

By Decebal Constantin Mocanu, Maria Torres Vega, Eric Eaton, Peter Stone, Antonio Liotta. 2016

27 citations recorded by the source corpus.

Conceived in the early 1990s, Experience Replay (ER) has been shown to be a successful mechanism to allow online learning algorithms to reuse past experiences. Traditionally, ER can be applied to all machine learning paradigms (i.e., unsupervised, supervised, and reinforcement learning). Recently, ER has contributed to improving the performance of deep reinforcement learning. Yet, its application to many practical settings is still limited by the memory requirements of ER, necessary to explicitly store previous observations. To remedy this issue, we explore a novel approach, Online Contrastive Divergence with Generative Replay (OCD_GR), which uses the generative capability of Restricted Boltzmann Machines (RBMs) instead of recorded past experiences. The RBM is trained online, and does not require the system to store any of the observed data points. We compare OCD_GR to ER on 9 real-world datasets, considering a worst-case scenario (data points arriving in sorted order) as well as a more realistic one (sequential random-order data points). Our results show that in 64.28% of the cases OCD_GR outperforms ER and in the remaining 35.72% it has an almost equal performance, while having a considerably reduced space complexity (i.e., memory usage) at a comparable time complexity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Management Science and Operations Research, Representation & Self-Supervised Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard E. Turner
2. Pratik Chaudhari
3. Rahul Ramesh
4. Fei Ye
5. Adrian G. Bors
6. Tameem Adel
7. Han Zhao
8. Mykola Pechenizkiy

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Richard E. Turner
3. Pratik Chaudhari
4. Rahul Ramesh
5. Trevor Darrell
6. Bernt Schiele
7. Ming-Hsuan Yang
8. Dacheng Tao

In Management Science and Operations Research

1. Pratik Chaudhari
2. Richard E. Turner
3. Rahul Ramesh
4. Tameem Adel
5. Decebal Constantin Mocanu
6. Han Zhao

In Representation & Self-Supervised Learning

1. Tameem Adel
2. Pratik Chaudhari
3. Rahul Ramesh
4. Hanul Shin
5. Richard E. Turner
6. Mykola Pechenizkiy
7. Jaehong Kim
8. Jung Kwon Lee

In Meta & Continual Learning

1. Richard E. Turner
2. Hanul Shin
3. Pratik Chaudhari
4. Mykola Pechenizkiy
5. Rahul Ramesh
6. Jaehong Kim
7. Jung Kwon Lee
8. Tameem Adel

Citations in the corpus

• Continual Learning with Deep Generative Replay (2017, methodology): “[24], in which they trained Restricted Boltzmann Machine to recover past input distribution.”
• Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…et al., 2020a), data replay (Robins, 1995; Shin et al., 2017; Lee et al., 2017), new architectures (Serra et al., 2018), generative replay-based (Mocanu et al., 2016; Shin et al., 2017; Liu et al., 2020; Ven et al., 2020), ensemble-based (Aljundi et al., 2017; Wen et al., 2020) and methods that…”
• Continual Learning with Adaptive Weights (CLAW) (2019, methodology): “In addition to the explicit replay form, some works have been based on generative replay (Thrun, 1996; Schmidhuber, 2013; Mocanu et al., 2016; Rebuffi et al., 2017; Kamra et al., 2017; Shin et al., 2017; van de Ven & Tolias, 2018; Wu et al., 2018).”
• SpaceNet: Make Free Space For Continual Learning (2020, methodology): “Similar work has been done by Mocanu et al. [31].”