Transformers are Meta-Reinforcement Learners
By Luckeciano Carvalho Melo. 2022
70 citations recorded by the source corpus.
The transformer architecture and variants presented remarkable success across many machine learning tasks in recent years. This success is intrinsically related to the capability of handling long sequences and the presence of context-dependent weights from the attention mechanism. We argue that these capabilities suit the central role of a Meta-Reinforcement Learning algorithm. Indeed, a meta-RL agent needs to infer the task from a sequence of trajectories. Furthermore, it requires a fast adaptation strategy to adapt its policy for a new task -- which can be achieved using the self-attention mechanism. In this work, we present TrMRL (Transformers for Meta-Reinforcement Learning), a meta-RL agent that mimics the memory reinstatement mechanism using the transformer architecture. It associates the recent past of working memories to build an episodic memory recursively through the transformer layers. We show that the self-attention computes a consensus representation that minimizes the Bayes Risk at each layer and provides meaningful features to compute the best actions. We conducted experiments in high-dimensional continuous control environments for locomotion and dexterous manipulation. Results show that TrMRL presents comparable or superior asymptotic performance, sample efficiency, and out-of-distribution generalization compared to the baselines in these environments.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Matthew Riemer
- Irina Rish
- Yoshua Bengio
- Doina Precup
- Khimya Khetarpal
- Tim Rocktaschel
- Joelle Pineau
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…Miao, Peng, Real, Levine, Le, Lee, and Faust, 2021b; Kirsch, Flennerhag, van Hasselt, Friesen, Oh, and Chen, 2022; Wan, Peng, and Gangwani, 2022; Melo, 2022; Nam, Sun, Pertsch, Hwang, and Lim, 2022) and multi-agent RL (Foerster et al., 2018a,1; Kim et al., 2021a; Al-Shedivat, Bansal, Burda,…”
- Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “Transformers have been shown to be effective for meta-RL on simple domains (Melo, 2022) and for learning RL algorithms (Laskin et al., 2022) from offline data.”