Measuring and regularizing networks in function space

By Ari S. Benjamin, David Rolnick, Konrad Paul Kording. 2018

168 citations recorded by the source corpus.

To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a $L^2$ Hilbert space. We examine how typical networks behave in this space, and compare how parameter $\ell^2$ distances compare to function $L^2$ distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the $L^2/\ell^2$ ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the $L^2$ distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through $L^2$-space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

Citations in the corpus

Continual Learning via Sequential Function-Space Variational Inference (2023, methodology): “The method subsequently proposed by Benjamin et al. (2019) involves comparing the current network with all previous versions of the network and on data from all past tasks instead of with only the most recent network on data from the current task.”
Task-Aware Information Routing from Common Representation Space in Lifelong Learning (2023, background): “Function Distance Regularization (FDR; Benjamin et al. (2018)), Dark Experience Replay (DER++; Buzzega et al. (2020)) and CLS-ER (Arani et al., 2022) leverage soft targets in addition to ground truth labels to enforce consistency regularization across previous and current model predictions.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, neutral): “Among them, regularization methods work by introducing additional constraints in the form of loss terms; they are designed to limit the amount of total change either in parameter space [42, 89, 19, 2] or functional space [47, 10].”
Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “We compare CVT with state-of-the-art and well-established Online CL baselines, including 11 rehearsal-based methods (ER [63], GEM [49], AGEM [14], GSS [4], FDR [8], HAL [13], ASERμ [67], ERT [10], RM [6], SCR [52], and CLS [5]), 2 methods leveraging Knowledge Distillation (iCaRL [62] and DER++ [9]).”
Continual Learning through Retrieval and Imagination (2022, background): “Rehearsal-based methods prevent catastrophic forgetting by replaying a subset of the stored exemplars of the previous tasks (Benjamin et al. 2019; Hou et al. 2019).”
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022, critical): “However, as discussed in [16, 46], parameter regularization methods empirically present relatively lower accuracy than the techniques in other categories, and might serve as a poor proxy for keeping the network output [8].”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, background): “Function Distance Regularization (FDR) (Benjamin et al., 2018) saves the network response at the task boundaries and adds a consistency loss on top of ER.”
Continual semi-supervised learning through contrastive interpolation consistency (2021, methodology): “distillation [33, 34, 35], or using the memory to train the model in an offline fashion [36].”
Continual Deep Learning by Functional Regularisation of Memorable Past (2020, methodology): “The challenge in using weight-regularisation lies in the fact that the exact values of the weights do not really matter due to parametric symmetries (Benjamin et al., 2018; Bishop, 2006).”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, background): “However, for DNNs a change in the weights is a poor proxy for the difference in the outputs (Benjamin et al., 2018).”
Continual learning with hypernetworks (2019, methodology): “This can be achieved using an output regularizer of the following form, where past outputs play the role of pseudo-targets (Robins, 1995; Li & Hoiem, 2018; Benjamin et al., 2018):”
On the distance between two neural networks and the stability of learning (2020, background): “For more work of that ilk, Benjamin et al. (2019) empirically test the use of Euclidean distance as a proxy for functional distance, and find the relationship non-trivial and difficult to interpret.”
Continual Learning with Lifelong Vision Transformer (2022, methodology): “Recently, various works have focused on the more difficult and realistic class-incremental learning (class-IL) [3, 9, 14, 20, 36, 61, 75, 78, 89, 94], where the network is evaluated on all classes observed during the training, without requiring the task identity.”
Effects of Auxiliary Knowledge on Continual Learning (2022, methodology): “To validate the effectiveness of our approach using auxiliary data during training, we compare our method with other CL methods based on rehearsal strategies: ER [41], GEM [25], A-GEM [26], iCaRL [32], FDR [42], GSS [29], HAL [43], DER [28] and vanilla DER++ [28].”
Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learning (2023, background): “Conversely, functional-regularization methods confine functional changes concerning certain ’anchor points’ in the function space [2, 52, 40].”
DDGR: Continual Learning with Deep Diffusion-based Generative Replay (2023, methodology): “There are also other regularization-based methods that are based on functional regularization (Pan et al., 2020; Benjamin et al., 2019), uncertainty regularization (Ahn et al., 2019), classifier projection regularization (Cha et al., 2021) and node importance (Jung et al., 2020).”