How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets

By Zhiyun Lu, Avner May, Kuan Liu, Alireza Bagheri Garakani, Dong Guo, Aurélien Bellet, Linxi (Jim) Fan, Michael Collins, Brian Kingsbury, Michael A. Picheny and 1 more. 2014

66 citations recorded by the source corpus.

The computational complexity of kernel methods has often been a major barrier for applying them to large-scale learning problems. We argue that this barrier can be effectively overcome. In particular, we develop methods to scale up kernel models to successfully tackle large-scale learning problems that are so far only approachable by deep learning architectures. Based on the seminal work by Rahimi and Recht on approximating kernel functions with features derived from random projections, we advance the state-of-the-art by proposing methods that can efficiently train models with hundreds of millions of parameters, and learn optimal representations from multiple kernels. We conduct extensive empirical studies on problems from image recognition and automatic speech recognition, and show that the performance of our kernel models matches that of well-engineered deep neural nets (DNNs). To the best of our knowledge, this is the first time that a direct comparison between these two methods on large-scale problems is reported. Our kernel methods have several appealing properties: training with convex optimization, cost for training a single model comparable to DNNs, and significantly reduced total cost due to fewer hyperparameters to tune for model selection. Our contrastive study between these two very different but equally competitive models sheds light on fundamental questions such as how to learn good representations.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A la Carte - Learning Fast Kernels (2014, background): “Lu et al. [2014] achieve promising performance on acoustic modelling problems, in some instances outperforming deep neural networks.”
Bayesian Nonparametric Kernel-Learning (2015, methodology): “However, naively applying MKL approaches would still require the computation of several large Gram matrices; instead, as previously discussed, one may combine random features to perform scalable MKL (see [15] for an application of such ideas).”
Steps Toward Deep Kernel Methods from Infinite Neural Networks (2015, methodology): “Such estimators were recently used within kernel methods to match deep learning results in language processing [13, 10].”
Bayesian Optimization with Tree-structured Dependencies (2017, methodology): “More specifically, we focus on the use of random Fourier features (Rahimi et al., 2007) that proved successful for large-scale kernel methods (Lu et al., 2014).”
Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy? (2015, result): “This also stands in line with the recent literature on convolutional kernel methods (see [47], [48]).”
A Survey of Modern Questions and Challenges in Feature Extraction (2015, background): “A successful attempt to solve that issue comes from the works that use distributed approximate feature maps (Lu et al., 2014; Huang et al., 2014), which allows them to build large scale kernel architectures that match deep networks in accuracy.”
Local Group Invariant Representations via Orbit Embeddings (2016, background): “The proposed approach can be suitable for largescale problems, benefiting from the recent advances in scalability of randomized kernel methods [3, 13, 25].”