Convolutional Kernel Networks

By Julien Mairal, Piotr Koniusz, Zaïd Harchaoui, Cordelia Schmid. 2014

402 citations recorded by the source corpus.

An important goal in visual recognition is to devise image representations that are invariant to particular transformations. In this paper, we address this goal with a new type of convolutional neural network (CNN) whose invariance is encoded by a reproducing kernel. Unlike traditional approaches where neural networks are learned either to represent data or for solving a classification task, our network learns to approximate the kernel feature map on training data. Such an approach enjoys several benefits over classical ones. First, by teaching CNNs to be invariant, we obtain simple network architectures that achieve a similar accuracy to more complex ones, while being easy to train and robust to overfitting. Second, we bridge a gap between the neural network literature and kernels, which are natural tools to model invariance. We evaluate our methodology on visual recognition tasks where CNNs have proven to perform well, e.g., digit recognition with the MNIST dataset, and the more challenging CIFAR-10 and STL-10 datasets, where our accuracy is competitive with the state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Making Convolutional Networks Shift-Invariant Again (2019, background): “Mairal et al. (2014) derive a network architecture, motivated by translation invariance, named Convolutional Kernel Networks.”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, neutral): “In convolutional network design, antialiasing has taken form in average pooling [43] and Gaussian filtering [49].”
Structured Convolutional Kernel Networks for Airline Crew Scheduling (2021, methodology): “This section contains additional details on CKNs (Mairal et al., 2014).”
Scattering Networks for Hybrid Representation Learning (2018, neutral): “CKN [35] 82.2 Roto-Scat + SVM [42] 82.3 ExemplarCNN [19] 84.3 DCGAN [45] 82.8 Scat + FC (ours) 84.7”
Deep Clustering for Unsupervised Learning of Visual Features (2018, methodology): “[39] learn patch level Convolutional Kernel Network [40] using an image retrieval setting.”
Unsupervised Learning by Predicting Noise (2017, background): “Several approaches have been recently proposed to tackle the problem of deep unsupervised learning (Coates & Ng, 2012; Mairal et al., 2014; Dosovitskiy et al., 2014).”
Stacked What-Where Auto-encoders (2015, background): “43% model accuracy Convolutional Kernel Networks [17] 62.”
Federated Multi-Task Learning (2017, background): “While MOCHA does not apply to non-convex deep learning models in its current form, we note that there may be natural connections between this approach and “convexified” deep learning models [6, 34, 51, 56] in the context of kernelized federated multi-task learning.”
A Unified Gradient Regularization Family for Adversarial Examples (2015, methodology): “We have achieved 39 errors on testing set, which ties with the recently proposed sophisticated convolutional kernel network described by [19].”
Convolutional Neural Fabrics (2016, neutral): “respectively, N denotes no data augmentation. Year Augmentation # Params. Error (%) Chang et al . [1] 2015 N 447K 0.24 Lee et al . [17] 2015 N 0.31 Wan et al . (Dropconnect) [32] 2013 T 379K 0.32 CKN [21] 2014 N 43 K 0.39 Goodfellow et al . (MaxOut) [4] 2013 N 420 K 0.45 Lin et al . (Network in Netw”
When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data (2017, background): “convolutional kernel networks [23], SVM neural networks [33]).”
Deep Fried Convnets (2014, methodology): “Our work is in line with recent interest in combining kernels with deep neural networks (Huang et al., 2014; Cho & Saul, 2009; Dai et al., 2014; Mairal et al., 2014).”
On Exact Computation with an Infinitely Wide Neural Net (2019, methodology): “, 2017] or use a neural network to extract features and then applying a kernel method on top of them [Mairal et al., 2014].”
SGD Learns the Conjugate Kernel Class of the Network (2017, background): “earn in polynomial time. Connection to kernels. As mentioned earlier, our paper builds on Daniely et al. [13], who developed the association of kernels to NN which we rely on. Several previous papers [24, 10,28,27,25,32,18,26,6,5,16,2] investigated such associations, but in a more restricted setting”
Biological Sequence Modeling with Convolutional Kernel Networks (2017, positive): “In this paper, we adopt a hybrid approach between kernel methods and deep neural networks, by adapting the convolutional kernel network (CKN) model originally developed in [27, 26] for image data.”
Tensor Representations via Kernel Linearization for Action Recognition from 3D Skeletons (2016, background): “Similarly, convolutional kernel networks [23] consider stacked layers of a variant of kernel descriptors [22].”