Learning Shared Representations in Multi-task Reinforcement Learning

By Diana Borsa, Thore Graepel, John Shawe-Taylor. 2016

48 citations recorded by the source corpus.

We investigate a paradigm in multi-task reinforcement learning (MT-RL) in which an agent is placed in an environment and needs to learn to perform a series of tasks, within this space. Since the environment does not change, there is potentially a lot of common ground amongst tasks and learning to solve them individually seems extremely wasteful. In this paper, we explicitly model and learn this shared structure as it arises in the state-action value space. We will show how one can jointly learn optimal value-functions by modifying the popular Value-Iteration and Policy-Iteration procedures to accommodate this shared representation assumption and leverage the power of multi-task supervised learning. Finally, we demonstrate that the proposed model and training procedures, are able to infer good value functions, even under low samples regimes. In addition to data efficiency, we will show in our analysis, that learning abstractions of the state space jointly across tasks leads to more robust, transferable representations with the potential for better generalization. this shared representation assumption and leverage the power of multi-task supervised learning. Finally, we demonstrate that the proposed model and training procedures, are able to infer good value functions, even under low samples regimes. In addition to data efficiency, we will show in our analysis, that learning abstractions of the state space jointly across tasks leads to more robust, transferable representations with the potential for better generalization.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Leveraging the assumption of shared common structure between tasks, Borsa, Graepel, and Shawe-Taylor (2016) build on the work of Calandriello, Lazaric, and Restelli (2014) and explicitly model a shared abstraction of the state-action space.”
Multi-Task Reinforcement Learning with Context-based Representations (2021, background): “(Calandriello et al., 2014; Borsa et al., 2016; Maurer et al., 2016) assume that in the multi-task settings, tasks share a common, low dimensional representation, and therefore advocate for learning a shared representation space across all the tasks.”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “By simultaneously learning about different tasks, the agent can exploit their common structure, which can lead to faster learning and better generalization to unseen tasks from the same family (Taylor & Stone, 2009; Lazaric, 2012; Ammar et al., 2012; 2014; Parisotto et al., 2015; Borsa et al., 2016;”
Transfer Reinforcement Learning with Shared Dynamics (2017, methodology): “[5] does not make any assumption of this kind, but their algorithm cannot be directly applied to an online learning problem.”
Hierarchy Through Composition with Multitask LMDPs (2017, methodology): “The Multitask LMDP To build a multitask action module, we exploit this compositionality to build a basis set of tasks (Foster & Dayan, 2002; Ruvolo & Eaton, 2013; Schaul et al., 2015; Pan et al., 2015; Borsa et al., 2016).”
Invariant Causal Prediction for Block MDPs (2020, background): “, 2017) and shared state-action representation (Borsa et al., 2016) to capture common or invariant behavior”