L2 Regularization for Learning Kernels

By Corinna Cortes, Mehryar Mohri, Afshin Rostamizadeh. Conference on Uncertainty in Artificial Intelligence, 2009

518 citations recorded by the source corpus.

The choice of the kernel is critical to the success of many learning algorithms but it is typically left to the user. Instead, the training data can be used to learn the kernel by selecting it out of a given family, such as that of non-negative linear combinations of p base kernels, constrained by a trace or L1 regularization. This paper studies the problem of learning kernels with the same family of kernels but with an L2 regularization instead, and for regression problems. We analyze the problem of learning kernels with ridge regression. We derive the form of the solution of the optimization problem and give an efficient iterative algorithm for computing that solution. We present a novel theoretical analysis of the problem based on stability and give learning bounds for orthogonal kernels that contain only an additive term O(√p/m) when compared to the standard kernel ridge regression stability bound. We also report the results of experiments indicating that L1 regularization can lead to modest improvements for a small number of kernels, but to performance degradations in larger-scale cases. In contrast, L2 regularization never degrades performance and in fact achieves significant improvements with a large number of kernels.

Subfields: Artificial Intelligence, Computational Mechanics

View Paper: https://www.semanticscholar.org/paper/61b0f5cd87a88ac98ab91239147157b92ecaade9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Algorithms for Learning Kernels Based on Centered Alignment (2012, methodology): “…parameter of the SVM. Norm-2 regularized combination ( l2-krr ): this algorithm optimizes the kernel ridge regression objective The L 2 regularized method is used for regression since it is shown in (Cortes et al., 2009a) to outperform the alternative L 1 regularized method in similar settings.”
Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks (2021, background): “For example, L2 regularization is a common way to penalize outlier weights with large absolute values, driving them close towards zero [18].”
A Survey of Modern Questions and Challenges in Feature Extraction (2015, methodology): “, 1998), if a linear combination of kernels is learned jointly, then the resulting supervised coupled problem requires solving following minimax problem min‖μ‖q≤1 maxα L(μ,α) = −λαTα− ∑p k=1 μkα Kkα+ 2α T y (Cortes et al., 2009).”
Learning Non-Linear Combinations of Kernels (2009, background): “The related problem of learning the kernel Kμ concomitantly can be formulated as the following min-max optimization problem [9]:”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “We hypothesize this is likely because `2 is generally a smoother metric [11, 36].”
Multiple Operator-valued Kernel Learning (2012, methodology): “This algorithm is similar to that of [8] and [15] both being based on alternating optimization.”
Simple and Efficient Multiple Kernel Learning by Group Lasso (2010, background): “A special case isL2-MKL (Kloft et al., 2008; Cortes et al., 2009a).”
Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN (2023, methodology): “This evolution comes from the designing of activation functions [1]–[3], proposing regularization methods of CNN parameters [4]–[6], constructing new optimization methods [7]–[9], cost functions [10]–[12], and new network architectures [13]–[17].”
Multiple Kernel Learning from Noisy Labels by Stochastic Programming (2012, methodology): “n the ﬁve data sets. We observe that for almost 2We do not compare many other MKL algorithms because (i) some algorithms (Xu et al., 2010) optimize the same objective as in (2), (ii) some algorithms (Cortes et al., 2009; Kloft et al., 2009) focus on diﬀerent regularizations (e.g. ℓ2 or p), and (iii)”
Insights from Classifying Visual Concepts with Multiple Kernel Learning (2011, background): “ickly as it can be wrapped around a regular support vector machine [15]. However, some evidence shows that sparse kernel mixtures are often outperformed by an unweighted-sum kernel [17]. As a remedy, [18, 19] propose ℓ2-norm regularized MKL variants, which promote non-sparse kernel mixtures and subs”
Multiple Kernel Learning Algorithms (2011, methodology): “Cortes et al. (2009) discuss the suitability of l2-norm for MKL.”