History Compression via Language Models in Reinforcement Learning
By Fabian Paischer, Thomas Adler, Vihang Patil, Angela Bitto-Nemling, Markus Holzleitner, Sebastian Lehner, Hamid Eghbalzadeh, Sepp Hochreiter. 2022
53 citations recorded by the source corpus.
In a partially observable Markov decision process (POMDP), an agent typically uses a representation of the past to approximate the underlying MDP. We propose to utilize a frozen Pretrained Language Transformer (PLT) for history representation and compression to improve sample efficiency. To avoid training of the Transformer, we introduce FrozenHopfield, which automatically associates observations with pretrained token embeddings. To form these associations, a modern Hopfield network stores these token embeddings, which are retrieved by queries that are obtained by a random but fixed projection of observations. Our new method, HELM, enables actor-critic network architectures that contain a pretrained language Transformer for history representation as a memory module. Since a representation of the past need not be learned, HELM is much more sample efficient than competitors. On Minigrid and Procgen environments HELM achieves new state-of-the-art results. Our code is available at https://github.com/ml-jku/helm.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Günter Klambauer
- Hubert Ramsauer
- Sepp Hochreiter
- Angela Bitto-Nemling
- Hamid Eghbalzadeh
- Fabian Paischer
- Fei Tang
- Vihang Patil
In Reinforcement Learning
- Hamid Eghbalzadeh
- Angela Bitto-Nemling
- Fabian Paischer
- Christian Alexander Steinparz
- Hubert Ramsauer
- Vihang Patil
- Marius-Constantin Dinu
- Thomas Schmied
Citations in the corpus
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “Modern Hopfield networks have already been successfully applied to immune repertoire classification (Widrich et al., 2020), chemical reaction prediction (Seidl et al., 2022) and reinforcement learning (Widrich et al., 2021; Paischer et al., 2022).”
- Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning (2022, background): “, 2022), abstraction (Shanahan & Mitchell, 2022; Paischer et al., 2022), and exploration (Sutton & Barto, 1998).”