Multiclass multiple kernel learning

By Alexander Zien, Cheng Soon Ong. International Conference on Machine Learning, 2007

310 citations recorded by the source corpus.

In many applications it is desirable to learn from several kernels. "Multiple kernel learning" (MKL) allows the practitioner to optimize over linear combinations of kernels. By enforcing sparse coefficients, it also generalizes feature selection to kernel selection. We propose MKL for joint feature maps. This provides a convenient and principled way for MKL with multiclass problems. In addition, we can exploit the joint feature map to learn kernels on output spaces. We show the equivalence of several different primal formulations including different regularizers. We present several optimization methods, and compare a convex quadratically constrained quadratic program (QCQP) and two semi-infinite linear programs (SILPs) on toy data, showing that the SILPs are faster than the QCQP. We then demonstrate the utility of our method by applying the SILP to three real world datasets.

Subfields: Artificial Intelligence, Computational Mechanics

View Paper: https://www.semanticscholar.org/paper/8db24719187e8a019a19360aa452583e43f1ba33

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Multiple Operator-valued Kernel Learning (2012, methodology): “ming existing methods. It would be interesting for future work to thoroughly compare the proposed MKL method for operator estimation with previous related methods for multi-class and multi-label MKL (Zien and Ong, 2007; Tang et al., 2009), in the contexts of structured output learning (Tsochantaridi”
View-Invariant Action Recognition Using Latent Kernelized Structural SVM (2012, methodology): “Following [22], the dual optimization problem of standard structural SVM is formulated by”
L2 Regularization for Learning Kernels (2009, background): “…of learning kernels has been investigated in a number of recent publications including (Lanckriet et al., 2004; Micchelli & Pontil, 2005; Argyriou et al., 2005; Ar-gyriou et al., 2006; Srebro & Ben-David, 2006; Ong et al., 2005; Lewis et al., 2006; Zien & Ong, 2007; Jebara, 2004; Bach, 2008).”
On feature combination for multiclass object classification (2009, background): “The authors of [22] derive such a MC-MKL formulation in which all parameters for all classes are trained jointly.”
A Binary Classification Framework for Two-Stage Multiple Kernel Learning (2012, methodology): “Multi-class multiple kernel learning (MC-MKL) of (Zien & Ong, 2007) is also close to our method and other baselines.”
Efficient and Accurate Lp-Norm Multiple Kernel Learning (2009, methodology): “Zien and Ong [ 27 ] showed that the MKL optimization problems by Bach et al. [3], Sonnenburg et al. [21], and their own formulation are equivalent.”
Optimization for machine learning (2010, methodology): “Here, we focus on the convex formulation of the MKL problem for classification as first stated in (Zien and Ong, 2007; Rakotomamonjy et al., 2007).”
Predicting abnormal returns from news using text classification (2008, methodology): “…probability in (1) by learning an optimal linear combination of kernels (Bousquet and Herrmann 2003, Bach et al. 2004, Lanckriet et al. 2004a, Ong et al. 2005, Sonnenberg et al. 2006, Micchelli and Pontil 2007, Zien and Ong 2007, Rakotomamonjy et al. 2008, Gönen and Alpaydin 2011).”
Learning The Discriminative Power-Invariance Trade-Off (2007, background): “There are many ways of converting distances to inner products and one is free to choose whichever embedding is most suitable.”
More generality in efficient multiple kernel learning (2009, methodology): “Other interesting approaches have been proposed in (Argyriou et al., 2005; Ong et al., 2005; Zien & Ong, 2007) and include Hy-perkernels and multi-class MKL. Note that these methods essentially learn linear combinations of base kernels subject to l 1 , or sometimes l 2 (Cristianini et al., 2001;…”
l p -Norm Multiple Kernel Learning (2011, background): “The prediction of the subcellular localization of proteins is one of the rare empirical success stories of l1-norm-regularized MKL (Ong and Zien, 2008; Zien and Ong, 2007): after defining 69 kernels that capture diverse aspects of protein sequences, l1-norm-MKL could raise the predictive accuracy si”
Simple and Efficient Multiple Kernel Learning by Group Lasso (2010, methodology): “Although this equivalence is not our invention, our derived variant equivalence not only leads to an efficient algorithm for MKL, but also generalizes to the case for Lp-MKL ( p ≥ 1 and denoting theLp-norm of kernel weights).”
Learning Output Kernels with Block Coordinate Descent (2011, background): “Previous works, such as (Zien & Ong, 2007; Lampert & Blaschko, 2008), have shown that learning convex combinations of kernels on the output space can improve performance.”
Learning Kernels with Random Features (2016, background): “…or classes of kernels and either uses heuristic rules to combine kernels [2] [23], optimizes structured (e.g. linear, nonnegative, convex) compositions of the kernels with respect to an alignment metric [9] [16] [20] [28], or jointly optimizes kernel compositions with empirical risk [17] [20] [29].”
Multiple Kernel Learning Algorithms (2011, unknown): “Zien and Ong (2007) develop a QCQP formulation and convert this formulation in two diﬀerent SILP problems for multiclass classiﬁcation problems.”