Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains
By Christian Gumbsch, Martin Volker Butz, Georg Martius. 2021
28 citations recorded by the source corpus.
A common approach to prediction and planning in partially observable domains is to use recurrent neural networks (RNNs), which ideally develop and maintain a latent memory about hidden, task-relevant factors. We hypothesize that many of these hidden factors in the physical world are constant over time, changing only sparsely. To study this hypothesis, we propose Gated $L_0$ Regularized Dynamics (GateL0RD), a novel recurrent architecture that incorporates the inductive bias to maintain stable, sparsely changing latent states. The bias is implemented by means of a novel internal gating function and a penalty on the $L_0$ norm of latent state changes. We demonstrate that GateL0RD can compete with or outperform state-of-the-art RNNs in a variety of partially observable prediction and control tasks. GateL0RD tends to encode the underlying generative factors of the environment, ignores spurious temporal dependencies, and generalizes better, improving sampling efficiency and overall performance in model-based planning and reinforcement learning tasks. Moreover, we show that the developing latent states can be easily interpreted, which is a step towards better explainability in RNNs.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Pablo Samuel Castro
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- Maxime Chevalier-Boisvert
- Sergey Levine
- Lucas Willems
In Statistical and Nonlinear Physics
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “However, it may still be the case that changes are limited to only a small subset of the causal variables of the underlying MDP (Gumbsch et al., 2021).”
- Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks (2023, methodology): “In Liu et al. [12], Miniworld is used to develop a new meta-learning approach that avoids local optima in end-to-end training, without sacrificing optimal exploration and Hutsebaut-Buysse et al. [8] explored the use of pre-trained task-independent word embedding for transfer learning.”