Online Multi-Task Learning for Policy Gradient Methods

By Haitham Bou-Ammar, Eric Eaton, Paul Ruvolo, Matthew E. Taylor. 2014

175 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “For example, Ammar et al. (2014) and Ammar et al. (2015) transfer knowledge via a latent basis whereas Brunskill and Li (2014) perform batch optimization across all encountered tasks.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “One approach (Wilson et al., 2007; Ammar et al., 2014) is to model the distribution over tasks and use this distribution as a prior when a new task is seen.”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, methodology): “For LPG-FTW and PG-ELLA, we ﬁxed all regularization parameters to 10 − 5 and the number of columns in L to k =5 , unless otherwise noted.”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “The multi-task literature [18, 19], lifelong learning literature [20, 21], and meta-learning literature [22, 23] describe ideas that are tightly coupled with this concept.”
Learning modular neural network policies for multi-task and multi-robot transfer (2016, methodology): “The PG-Ella algorithm uses policy gradients for sequential multitask learning [17].”
Learning to Generalize: Meta-Learning for Domain Generalization (2017, neutral): “…to changes in its operating environment (MDP) – without being allowed any rewarded ( ≈ supervised domain adaptation (Finn, Abbeel, and Levine 2017; Ammar et al. 2014)), or un-rewarded ( ≈ unsupervised domain adaptation (Finn et al. 2017; Ammar et al. 2015b)) trials in the target environment…”
Benchmark Environments for Multitask Learning in Continuous Domains (2017, background): “…a simple mass spring damper task, cart-pole with continuous control; a three-link inverted pendulum with continuous control; a quadrotor control task (Ammar et al., 2014); a double-linked pendulum task; a modiﬁed cartpole balancing task which can transfer to physical system (Higuera et al., 2017).”
Learning to Learn: Meta-Critic Networks for Sample Efficient Learning (2017, methodology): “In this experiment, we exclude All+FT as it performed much worse than the others and include the PG-ELLA [1] method (lifelong policy gradient-based reinforcement learning via a low-rank policy factorisation), which we extended to work with a neural network policy as per [39].”
NerveNet: Learning Structured Policy with Graph Neural Networks (2018, methodology): “In Ammar et al. (2014), a multi-task policy gradient is proposed, while, Calandriello et al. (2014) propose multi-task extensions of the fitted Q-iteration algorithm. Successor features in Barreto et al. (2017) are used to boost the performance of multiple tasks. Unlike ours, these methods do not ex”
Autonomous Cross-Domain Knowledge Transfer in Lifelong Policy Gradient Reinforcement Learning (2015, methodology): “Figures (a) and (b) depict task domains where cross-domain transfer has a significant impact, showing that our approach outperforms standard PG and PG-ELLA.”
Using Task Descriptions in Lifelong Machine Learning for Improved Performance and Zero-Shot Transfer (2017, methodology): “This was extended to handle reinforcement learning by Bou Ammar et al. (2014) via approximating the RL multi-task objective by ﬁrst substituting in the convex lower-bound to the PG objective J ( α ( t ) ) in order to make the optimization convex.”
Source Task Creation for Curriculum Learning (2016, background): “For example, 3v2 keepaway serving as a source for 4v3 keepaway [26], 2D mountain car as a source for 3D mountain car [25], or varying parameters of physical systems [2].”
Online Multitask Relative Similarity Learning (2017, unknown): “In [Ammar et al. , 2014; Calandriello et al. , 2014], these ELLA algorithms were used for reinforcement learning.”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, background): “For example, Ammar et al. (2014) and Mendez et al. (2020) assign task-speciﬁc parameters to each task while sharing a reusable knowledge base among all tasks.”
Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning (2017, methodology): “Most previous MRL approaches assumed access to data from all tasks (Bou-Ammar et al. 2014; Parisotto, Ba, and Salakhutdinov 2016; Teh et al. 2017).”
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, background): “This is typically done by sharing a representation among tasks, using distillation or a latent basis [18].”
Multitask Soft Option Learning (2019, background): “e a KLdivergence between them for training. However, they do not consider termination probabilities and instead only choose one option per task. Instead of transferring option policies between tasks, Ammar et al. (2014) aim to share behavior through a latent embedding. Another interesting approach t”