Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

By Johan Ferret, Raphaël Marinier, Matthieu Geist, Olivier Pietquin. 2019

32 citations recorded by the source corpus.

The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample-efficient. Our main contribution is SECRET, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Many works have studied better credit assignment via state-association, learning an architecture which decomposes the reward function such that certain “important” states comprise most of the credit [45, 46, 15].”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “Although this format is convenient for some algorithms [36], it discards the temporal information initially present in the data and prevents building methods exploiting such information [34, 24, 25, 9, 14].”
Agent57: Outperforming the Atari Human Benchmark (2020, background): “…two types: ensuring that gradients correctly assign credit (Ke et al., 2017; Weber et al., 2019; Fortunato et al., 2019) and using values or targets to ensure correct credit is assigned (Arjona-Medina et al., 2019; Hung et al., 2019; Liu et al., 2019; Harutyunyan et al., 2019; Ferret et al., 2020).”
Synthetic Returns for Long-Term Credit Assignment (2021, methodology): “Other recent work has used attention weights as a proxy for credit: Ferret et al. (2019) apply this approach to improve transfer, while (Liu et al., 2019) focus on artiﬁcial delayed-reward problems using sequence-wide prediction.”