Interference and Generalization in Temporal Difference Learning

By Emmanuel Bengio, Joelle Pineau, Doina Precup. 2020

69 citations recorded by the source corpus.

We study the link between generalization and interference in temporal-difference (TD) learning. Interference is defined as the inner product of two different gradients, representing their alignment. This quantity emerges as being of interest from a variety of observations about neural networks, parameter sharing and the dynamics of learning. We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning. We hypothesize that the cause can be traced back to the interplay between the dynamics of interference and bootstrapping. This is supported empirically by several observations: the negative relationship between the generalization gap and interference in TD, the negative effect of bootstrapping on interference and the local coherence of targets, and the contrast between the propagation rate of information in TD(0) versus TD($\lambda$) and regression tasks such as Monte-Carlo policy evaluation. We hope that these new findings can guide the future discovery of better bootstrapping methods.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Meanwhile, untrained agents often need to play far more of these games over their lifetime than their human competition and even after doing so, lack the ability to generalize to new variations even for simple RL problems (Bengio, Pineau, and Precup, 2020).”
MOPO: Model-based Offline Policy Optimization (2020, background): “However, such online data collection can incur high sample complexity [45, 55, 56], limit the power of generalization to unseen random initialization [8, 74, 4], and pose risks in safety-critical settings [67].”
Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation (2021, background): “2) Interference [22]: It is a type of inﬂuence between two gradient-based processes with objectives J 1 and J 2 and sharing parameter θ .”
Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation (2021, methodology): “One downside of the proposed method is that, as for TD-based methods, the use of bootstrapping may cause optimization challenges (Kumar et al., 2020; Bengio et al., 2020) and limit its performance.”