Using Noise to Compute Error Surfaces in Connectionist Networks: A Novel Means of Reducing Catastrophic Forgetting

By R. M. French, N. Chater. 2002

71 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

DER: Dynamically Expandable Representation for Class Incremental Learning (2021, background): “In detail, excessive plasticity often causes large performance degradation of the old categories, referred to as catastrophic forgetting [8].”
Continual Learning With Extended Kronecker-Factored Approximate Curvature (2020, methodology): “This approximation of the multitask learning objective was used for Hessian pseudopattern backpropagation [7], and it is actually the same as elastic weight consolidation (EWC) [15] and online structured Laplace approximation (OSLA) [31] except that they are grounded in Bayesian approaches.”
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections (2020, background): “gm is well-established to be the dominant approach [Raffel et al., 2019], as training multiple tasks using a single set of parameters can be problematic in many ways, such as catastrophic forgetting [French and Chater, 2002, McCloskey and Cohen, 1989, McClelland et al., 1995, Kirkpatrick et al., 201”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, background): “When learning from data containing diverse patterns, catastrophic forgetting [8, 25] is observed when attempting to perform multi-task or transfer learning by fine-tuning.”
Critical periods and catastrophic interference effects in the development of self-organizing feature maps. (2008, methodology): “Alternatively, noise or ‘pseudo’ patterns may be used in order to extract the function learned by the network in response to early training (French & Chater, 2002).”
HyperGrid Transformers: Towards A Single Model for Multiple Tasks (2021, background): “, 2019), as training multiple tasks using a single set of parameters can be problematic in many ways, such as catastrophic forgetting (French and Chater, 2002; McCloskey and Cohen, 1989; McClelland et al., 1995; Kirkpatrick et al., 2017) or the inherent difficulty of finding a consistently good mode”
Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods (2004, unknown): “In a detailed formal analysis French and Chater [23] generalise the concept pseudoitem to the point that explicit error information is no longer required, and error surfaces can be approximated (using ﬁrst–derivative information) purely from noise.”
Online Task-free Continual Learning with Dynamic Sparse Distributed Memory (2022, background): “…to a second, independent network and then inter-leaved with the new information being learned [16, 4], creating an approximation of the error surface of the previously learned patterns and integrating that error surface with the error surface associated with the new patterns [18], and so on.”