Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

By Tom Schaul, Diana Borsa, Joseph Modayil, Razvan Pascanu. 2019

79 citations recorded by the source corpus.

Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms. We study the learning dynamics of reinforcement learning (RL), specifically a characteristic coupling between learning and data generation that arises because RL agents control their future data distribution. In the presence of function approximation, this coupling can lead to a problematic type of 'ray interference', characterized by learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn one thing at a time even when learning in parallel is better. We establish the conditions under which ray interference occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting. We characterize a number of its properties and discuss possible remedies.

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Furthermore the multi-task regime is particularly suitable for the investigation of strongly off-policy data distributions arising from sharing the replay across agents, as concurrently learning agents can easily be stuck in different policy plateaus, generating substantially different data (Schaul ”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “By its design, it does not suffer from forgetting, however it is considered to be hard as multiple task “compete for attention of a single learning system”, see [20, 41].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “This is known to be difficult, to the point that often times learning completely separate policies for each of the tasks is actually faster, and results in better performance, than learning the tasks together (Yu et al., 2019; Schaul et al., 2019).”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “A number of recent works have studied algorithmic improvements in the area of multi-task reinforcement learning, as well as potential explanations for the difficulty of RL in the multi-task setting [8, 57].”
Benchmarking Model-Based Reinforcement Learning (2019, background): “[42] Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu.”
Gradient Surgery for Multi-Task Learning (2020, background): “Similarly to our work, a number of prior approaches have observed the difficulty of optimization in multi-task learning [26, 29, 52, 55].”
Constrained Reinforcement Learning Has Zero Duality Gap (2019, background): “Moreover, implicit interference between the goals may lead to training plateaus as they compete for resources in the policy [10].”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, background): “However, to realize these beneﬁts for a real-world robotic learning system, we need to overcome a number of major challenges [64, 32, 11, 86], which have so far made it difﬁcult to produce a large-scale demonstration of multi-task image-based RL that effectively accelerates the acquisition of…”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “…multiple tasks at once can present a challenge even for supervised learning problems due to different tasks being learned at different rates (Chen et al. 2018; Schaul et al. 2019) or the challenges in determining how to resolve conﬂicting gradient signals between tasks (Sener and Koltun 2018).”
Grounding Language in Play (2020, background): “RL additionally faces signiﬁcant optimization challenges (Schaul et al., 2019) in the multitask setting, often leading to worse data efﬁciency than learning tasks individually (Parisotto et al., 2015).”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “Prior work have studied the optimization challenges of multi-task training (Hes-sel et al., 2019; Schaul et al., 2019), while others suggest to improve training quality through learning task relatedness (Zhang & Yeung, 2012), routing task-specifc paths (Rusu et al., 2016; Rosenbaum et al., 2019),…”
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation (2021, background): “Researchers in DeepMind studied the learning dynamics of the single-task RL and developed a hypothesis that the characteristic coupling between learning and data generation is the main cause of interference and performance plateaus in deep RL systems [23].”