Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning

By Ali Rahimi, B. Recht. 2008

866 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

Do ImageNet Classifiers Generalize to ImageNet? (2019, neutral): “These are random features for CIFAR-10 (Coates et al., 2011; Rahimi and Recht, 2009) and Fisher vectors for ImageNet (Perronnin et al.”
Do CIFAR-10 Classifiers Generalize to CIFAR-10? (2018, methodology): “In addition, we also evaluated “shallow” approaches based on random features [2, 16].”
The Variational Fair Autoencoder (2015, methodology): “Furthermore the label information will also better guide the feature extraction towards the more salient parts of the data, thus maintaining most of the (predictive) information.”
On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions (2015, neutral): “y of eigenvalue is not fast, i.e., very close to 1/i, then we may only use the bound d(λ) = trΣ(Σ+λI)−1 6λ−1 trΣ = R2/λ, and thus a sufﬁcient condition n>10mlog2m, and we obtain the same result as Rahimi and Recht (2009). However, when we have eigenvalue decays as R2i−2s, we get (up to constants)”
Fastfood: Approximate Kernel Expansions in Loglinear Time (2014, methodology): “• For each observation x compute Mx and apply a nonlinearity ψ to each coordinate separately, i.e. (Rahimi and Recht, 2009) showed that performance was very competitive with conventional RBF kernel approaches while providing dramatically simpliﬁed code.”
FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test (2014, background): “Two of the representative developments are Random Kitchen Sinks (Rahimi & Recht, 2007; 2008) and Fastfood (Le et al., 2013), which can significantly speed up the computation for a large range of kernel functions by mapping data into a relatively lowdimensional randomized feature space.”
Scalable Kernel Methods via Doubly Stochastic Gradients (2014, background): “Thus the required number of random feature (or ranks), r, will be of the order O(n) = O(1/ ) [4, 5, 8, 9].”
Improving Multi-Step Prediction of Learned Time Series Models (2015, background): “Recent work in kernel machines enables us to instead approximate φ through generating random Fourier features (Rahimi and Recht 2007; 2008; Le, Sarl´os, and Smola 2013; L´azaro-Gredilla 2010).”
Diverse Neural Network Learns True Target Functions (2016, methodology): “cord with the practice of using gradient descent to learn neural networks. 2 Related work Kernel methods have many commonalities with one-hidden-layer neural networks. The random feature perspective [Rahimi and Recht, 2009, Cho and Saul, 2009] views kernels as linear combinations of nonlinear basis ”
A la Carte - Learning Fast Kernels (2014, methodology): “Rahimi and Recht [2009] suggested to sample from the spectral distribution ρ ( ω ) for a Monte Carlo approximation to the integral in (2).”
SGD Learns the Conjugate Kernel Class of the Network (2017, background): “earn in polynomial time. Connection to kernels. As mentioned earlier, our paper builds on Daniely et al. [13], who developed the association of kernels to NN which we rely on. Several previous papers [24, 10,28,27,25,32,18,26,6,5,16,2] investigated such associations, but in a more restricted setting”
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks (2019, methodology): “In addition, we define the (1)Since random feature models and kernel methods are highly related [31, 32], we group them into the same category.”
Steps Toward Deep Kernel Methods from Infinite Neural Networks (2015, methodology): “ithms to learn inﬁnite neural networks [2]. This work formulates learning an inﬁnite network as an inﬁnite convex program and devise an incremental algorithm that is based on its dual representation. [18] suggest to optimize an inﬁnite networks with a single layer using randomization to decrease the”
ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels (2019, methodology): “Rahimi and Recht (2008) proposed a random transform for approximating kernels for kernel methods (see also Rahimi and Recht 2009).”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “Random “kitchen sink” features have become a seminal approach in the machine learning literature (Rahimi & Recht, 2008; 2009).”
Semantic Pooling for Complex Event Analysis in Untrimmed Videos (2016, methodology): “For translation-invariant kernels (e.g., Gaussian), we can approximately derive from the kernel an explicit, ﬁnite dimensional, and nonlinear feature transformation f ð(cid:6)Þ [65].”