A la Carte - Learning Fast Kernels

By Zichao Yang, Andrew Gordon Wilson, Alex Smola, Le Song. 2014

139 citations recorded by the source corpus.

Kernel methods have great promise for learning rich statistical representations of large modern datasets. However, compared to neural networks, kernel methods have been perceived as lacking in scalability and flexibility. We introduce a family of fast, flexible, lightly parametrized and general purpose kernel learning methods, derived from Fastfood basis function expansions. We provide mechanisms to learn the properties of groups of spectral frequencies in these expansions, which require only O(mlogd) time and O(m) memory, for m basis functions and d input dimensions. We show that the proposed methods can learn a wide class of kernels, outperforming the alternatives in accuracy, speed, and memory consumption.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical Learning Theory

Citations in the corpus

Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages (2015, methodology): “To make the algorithm computationally tractable, we regress directly in the primal from random Fourier features of the data (Rahimi and Recht, 2007; Le et al., 2013; Yang et al., 2015).”
Compression of fully-connected layer in neural network by Kronecker product (2015, methodology): “[30] used adaptive fastfood transform to reparameterize the matrix-vector multiplication of fully-connected layers.”
Deep Fried Convnets (2014, background): “Each of the S,G andB parameters plays a different role in the Fastfood approximation (Yang et al., 2014):”
Stochastic Variational Deep Kernel Learning (2016, background): “ point. Gaussian processes are thus typically limited to at most a few thousand training points. Many promising approaches to scalability have been explored, for example, involving randomized methods [22, 18, 33] , and low rank approximations [25, 21]. Wilson and Nickisch [29] recently introduced th”
Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) (2015, methodology): “, 2010), a recent state of the art scalable GP method based on random projections with O(mn) computations and O(m) storage for m basis functions and n training points (see also Rahimi and Recht, 2007; Le et al., 2013; Yang et al., 2015; Lu et al., 2014).”
Thoughts on Massively Scalable Gaussian Processes (2015, methodology): “Moreover, inducing point and ﬁnite basis expansions (e.g., Qui˜nonero-Candela and Rasmussen, 2005a; L´azaro-Gredilla et al., 2010; Yang et al., 2015) cost O ( m ) and O ( m 2 ) per test point.”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “, 2018) and Fastfood approximate kernels (FF)5 (Yang et al., 2015).”
Randomness in neural networks: an overview (2017, methodology): “Their work is generalized in the framework presented by Yang et al.66 Along the same lines, a state-of-the-art solver for kernel methods on large-scale scenarios is presented in Dai et al.67 by combining RFFs with a stochastic evaluation of the overall gradient.”
Learning to Learn Kernels with Variational Random Features (2020, unknown): “Instead of modeling the spectral distribution with explicit density functions, other works focus on optimizing the random base sampling strategy (Yang et al., 2015; Sinha & Duchi, 2016).”
Understanding Probabilistic Sparse Gaussian Process Approximations (2016, methodology): “ifferently, e.g., function values at certain input locations, properties of the spectral representations [2], or more abstract representations [3]. Similar ideas are used in random feature expansions [4, 5]. Here we focus on methods that represent the approximate posterior using the function value a”
Bayesian Nonparametric Kernel-Learning (2015, methodology): “We note although the optimization of the AlaC problem above took on average nearly twice as long as our sampling, we achieved a lower MSE on a majority of the datasets.”
Proposal : Scalable , Active and Flexible Learning on Distributions (2015, unknown): “Yang et al. (2015) recently proposed a spectral kernel learning method capable of learning richer kernels than simple linear combinations or bandwidth selection; integrating that method into the mean map kernel, for two-sample testing as well as for machine learning models, could prove fruitful in i”
Large-scale Online Kernel Learning with Random Feature Reparameterization (2017, background): “Another solution, named à la carte [Yang et al., 2015], is to use gradient descent to maximize the marginal likelihood of a Gaussian process (GP) that is formalized for kernel learning via Fastfood basis function expansions.”