The Utility of Sparse Representations for Control in Reinforcement Learning

By Vincent Liu, Raksha Kumaraswamy, Lei Le, Martha White. 2018

68 citations recorded by the source corpus.

We investigate sparse representations for control in reinforcement learning. While these representations are widely used in computer vision, their prevalence in reinforcement learning is limited to sparse coding where extracting representations for new data can be computationally intensive. Here, we begin by demonstrating that learning a control policy incrementally with a representation from a standard neural network fails in classic control domains, whereas learning with a representation obtained from a neural network that has sparsity properties enforced is effective. We provide evidence that the reason for this is that the sparse representation provides locality, and so avoids catastrophic interference, and particularly keeps consistent, stable values for bootstrapping. We then discuss how to learn such sparse representations. We explore the idea of Distributional Regularizers, where the activation of hidden nodes is encouraged to match a particular distribution that results in sparse activation across time. We identify a simple but effective way to obtain sparse representations, not afforded by previously proposed strategies, making it more practical for further investigation into sparse representations for reinforcement learning.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computational Theory and Mathematics, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computational Theory and Mathematics

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Another way to understand modularity is from the perspective of sparsity, which has been explored extensively from a compression and ef ﬁ ciency angle [49 – 54] as well as for its applicability to reinforcement learning and control [55].”
Meta-Learning Representations for Continual Learning (2019, methodology): “SR-NN use the Set-KL method to learn a sparse representation (Liu et al., 2019) on the meta-training set.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “For instance, (Liu et al., 2018) point out that sparse representations may help Q-learning algorithms, which links back to prior literature on state-aliasing and destructive interference.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, background): “The ﬁrst insight was also demonstrated in (Liu et al., 2018) where enforcing sparsity in the Q-function was found to provide locality in the Q-function which prevented catastrophic interference and provided better values for bootstrapping.”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “We note that feature sparsity has been utilized to mitigate catastrophic forgetting for decades (McCloskey & Cohen, 1989; French, 1991; Liu et al., 2019; Lan & Mahmood, 2023).”
SPeCiaL: Self-Supervised Pretraining for Continual Learning (2021, methodology): “The idea of using sparse representations has been proposed in earlier work [French, 1991], and revisited in modern CL methods [Liu et al. , 2019; Aljundi et al. , 2018].”
Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (2024, methodology): “There are also sparsity-inducing methods (e.g., Liu et al. 2019) that work by maintaining sparse connections so that the weight updates can be localized and not affect many prior useful weights.”
Interference and Generalization in Temporal Difference Learning (2020, background): “…learning (Lopez-Paz & Ranzato, 2017; Schaul et al., 2019), it forms the Neural Tangent Kernel (Jacot et al., 2018), it is the Taylor expansion around θ of a gradient update (Achiam et al., 2019), as well as the Taylor expansion of pointwise loss differences (Liu et al., 2019b; Fort et al., 2019).”