Multi-Task Reinforcement Learning with Context-based Representations

By Shagun Sodhani, Amy Zhang, Joelle Pineau. 2021

259 citations recorded by the source corpus.

The benefit of multi-task learning over single-task learning relies on the ability to use relations across tasks to improve performance on any single task. While sharing representations is an important mechanism to share information across tasks, its success depends on how well the structure underlying the tasks is captured. In some real-world situations, we have access to metadata, or additional information about a task, that may not provide any new insight in the context of a single task setup alone but inform relations across multiple tasks. While this metadata can be useful for improving multi-task learning performance, effectively incorporating it can be an additional challenge. We posit that an efficient approach to knowledge transfer is through the use of multiple context-dependent, composable representations shared across a family of tasks. In this framework, metadata can help to learn interpretable representations and provide the context to inform which representations to compose and how to compose them. We use the proposed approach to obtain state-of-the-art results in Meta-World, a challenging multi-task benchmark consisting of 50 distinct robotic manipulation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Current work also typically assumes a stationary task state distribution for meta-learning (Doshi-Velez and Konidaris, 2013; Wang et al., 2016; Duan et al., 2016; Zintgraf et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2019; Humplik et al., 2019; Fakoor et al., 2019; Ortega et al., 2019; Perez”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “Deep neural networks [27, 28] have achieved impressive results in manipulation for single tasks [29], and recently led to more challenging setups such as multi-task learning [30, 31, 32, 33].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “Contemporary work demonstrates the utility of language commands for multi-task RL [29], while our work expands language-conditioning to encompass both IL and RL.”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Recent work focuses on learning a single multi-task policy with both onpolicy [34] and off-policy RL methods [35, 36, 37], and have shown promising results on a variety of robot manipulation tasks in simulation [34].”
Block Contextual MDPs for Continual Learning (2021, methodology): “This additional relaxation of the original CMDP definition as a form of Block MDP (Du et al., 2019) was previously introduced in Sodhani et al. (2021b) for the multi-task setting where the agent focuses on a subset of the whole space for a specific task, which we again present here for clarity: Defi”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “[76] found that language-conditioning leads to performance gains for multi-task RL on 50 task variations.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Sodhani et al. [250] proposed an open-source OpenNEED consisting of a large-scale, high-frame-rate non-eye (head, hand, and scene) and eye (3D gaze vector) data set.”
Conflict-Averse Gradient Descent for Multi-task Learning (2021, methodology): “Following [33], we use Soft Actor-Critic (SAC) [10] as the underlying RL training algorithm.”