Kernel Methods for Deep Learning

By Youngmin Cho, Lawrence K. Saul. 2009

865 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Wide neural networks of any depth evolve as linear models under gradient descent (2019, methodology): “[46] Youngmin Cho and Lawrence K Saul.”
Scaling description of generalization with number of parameters in deep learning (2019, background): “udy the ﬂuctuations of ft N throughout training for large networks usingtheNeuralTangentKernel[22]orNTK.Atinitializationt= 0,ft=0 N isa randomfunctionwhoselimitingdistributionasN!1isknowntobeGaussian [34,7,28]. ThesetypesofﬂuctuationsdonotvanishasN!1: thevariance offt=0 N atinitializationisessential”
Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018, methodology): “These kernels are used in Bayesian inference or Support Vector Machines, yielding results comparable to ANNs trained with gradient descent [9, 2].”
Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018, background): “We show that the double descent risk curve also appears with Random ReLU feature networks [12].”
Neural-net-induced Gaussian process regression for function approximation and PDE solution (2018, background): “For the ReLU nonlinearity, the base kernel reads [11] k1(x, x′) = 1 2π ‖x‖2‖x′‖2(sin θ0 + (π − θ0) cos θ0), (12)”
On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions (2015, background): “Alternatively, kernels based on neural networks with random weights are directly in the form of random features (Cho and Saul, 2009; Bach, 2014): for example, the kernel k ( x, y ) = E ( v ⊤ x ) s + ( v ⊤ x ) s + for v uniformly distributed in the hyper-sphere corresponds to sampling weights in a…”
Deriving Neural Architectures from Sequence and Graph Kernels (2017, background): “Our work follows recent work demonstrating the connection between neural networks and kernels (Cho & Saul, 2009; Hazan & Jaakkola, 2015).”
Deep Fried Convnets (2014, methodology): “Our work is in line with recent interest in combining kernels with deep neural networks (Huang et al., 2014; Cho & Saul, 2009; Dai et al., 2014; Mairal et al., 2014).”
Approximate Inference Turns Deep Networks into Gaussian Processes (2019, background): “Subsequently, many works derived explicit expressions for the GP kernel corresponding to neural networks [4, 7, 16] and their deep variants [5, 6, 13, 18].”
Calibrating Deep Convolutional Gaussian Processes (2018, methodology): “…the construction of a Bayesian linear model to approximate the GP latent variables as In this work, we focus in particular on the order-one ARC - COSINE kernel [3] k (1) arc where θ = ( σ, Λ = diag( 21 , . . . , 2 d )) and α is the angle between Λ − 12 c ( x i | Ψ ) and Λ − 12 c ( x j | Ψ ) .”
Deep Neural Networks as Gaussian Processes (2017, methodology): “They are exactly the relations derived in the mean ﬁeld theory of signal propagation in fully-connected random neural networks (Poole et al. (2016); Schoenholz et al. (2017)) and also appear in the literature on compositional kernels (Cho & Saul (2009); Daniely et al. (2016)).”
On Lazy Training in Differentiable Programming (2018, background): “When σ ( s ) = max { s, 0 } is the rectiﬁed linear unit activation and the distribution of the weights a j is rotation invariant in R d , one has the following explicit formulae [8]: where ϕ ∈ [0 , π ] is the angle between the two vectors x and x (cid:48) .”
On Exact Computation with an Infinitely Wide Neural Net (2019, background): “Cho and Saul [2009] derived a closed-form kernel formula for rectiﬁed polynomial activations, which include ReLU as a special case.”
Scalable Kernel Methods via Doubly Stochastic Gradients (2014, background): “of the input xparametrized by , or ˚ !(x) = [˚ ! 1 (x);˚ ! 2 (x);:::;˚ ! r (x)], where ˚ ! 1 (x) denote the random feature for kernel k 1(x;x0). The former random features dene a hierachical kernel [45], and the latter random features induce a linear combination of multiple kernels. It is worth to n”
Breaking the Curse of Dimensionality with Convex Neural Networks (2014, neutral): “Given the angle ϕ ∈ [0, π] defined through x ⊤x′ R2 + 1 = (cosϕ) √ ‖x‖22 R2 + 1 √ ‖x′‖22 R2 + 1, we have explicit expres- sions (Le Roux and Bengio, 2007; Cho and Saul, 2009): k0(z, z ′) = 1 2π (π − ϕ) k1(z, z ′) = √ ‖x‖22 R2 + 1 √ ‖x′‖22 R2 + 1 2(d + 1)π ((π − ϕ) cosϕ+ sinϕ) k2(z, z ′) = ( ‖x‖22 R2”
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation (2019, methodology): “Daniely et al. (2016) also proved similar results in the framework of kernel methods, where they introduced a notion of “computational skeleton,” similar to tensor programs introduced here, that covers feedforward computation with no weight-sharing (so that, for example, it can express locally conne”