Deep Learning using Linear Support Vector Machines

By Yichuan Tang. 2013

939 citations recorded by the source corpus.

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Deeply-Supervised Nets (2014, methodology): “Our experiments are primarily based on L2SVM objectives, previously used in DL by [27] for the output layer only; we also show that our approach of introducing companion objectives is not dependent on the use of L2SVM objectives — when using softmax we find similar performance improvements.”
Challenges in representation learning: A report on three machine learning contests (2013, background): “ unclear whether the performance of the best deep network has reached the Bayes rate on this task or not. 8 http://nnforge.org 5 Table 1. Private test set accuracy on FER-13 Team Members Accuracy RBM [16] Yichuan Tang 71.162% Unsupervised Yingbo Zhou, Chetan Ramaiah 69.267% Maxim Milakov8 Maxim Mila”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “erceptron (MLP) we train on MNIST consists of 3 hidden layers of 4096 binary units and a L2-SVM output layer; L2-SVM has been shown to perform better than Softmax on several classication benchmarks (Tang, 2013; Lee et al., 2014). We regularize the model with Dropout (Srivastava et al., 2014). The sq”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, methodology): “The MLP we train on MNIST consists in 3 hidden layers of 1024 Rectifier Linear Units (ReLU) [34, 24, 3] and a L2-SVM output layer (L2-SVM has been shown to perform better than Softmax on several classification benchmarks [30, 32]).”
Towards Deep Neural Network Architectures Robust to Adversarial Examples (2014, methodology): “However, performance is suboptimal compared to state-of-the-art DNNs, possibly due to lack of end-to-end training Vinyals et al. (2012); Tang (2013).”
Robust Large Margin Deep Neural Networks (2016, methodology): “The training of DNN by promoting a large output margin has also been explored empirically in [19].”
Recent advances in convolutional neural networks (2015, background): “The optimum parameters for a speciﬁc task can be obtained by minimizing an appropriate loss function deﬁned on that task.”
Multi-Task CNN Model for Attribute Prediction (2015, background): “Another work [51] confirms their finding that using the well-known SVM squared hinge loss does improve the final performance after training the CNN.”
BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 (2016, methodology): “The MLP we train on MNIST consists in 3 hidden layers of 4096 binary units (see Section 1) and a L2-SVM output layer; L2-SVM has been shown to perform better than Softmax on several classification benchmarks (Tang, 2013; Lee et al., 2014).”
Contractive Rectifier Networks for Nonlinear Maximum Margin Classification (2015, result): “Experimental results on MNIST, CIFAR-10, CIFAR-100 and MIT67 datasets demonstrate that the proposed contractive rectifier networks consistently outperform their conventional unconstrained rectifier network counterparts.”
On Loss Functions for Deep Neural Networks in Classification (2017, background): “L1 L1 loss ‖y − o‖1 L2 L2 loss ‖y − o‖(2)2 L1 ◦ σ expectation loss ‖y − σ(o)‖1 L2 ◦ σ regularised expectation loss(1) ‖y − σ(o)‖(2)2 L∞ ◦ σ Chebyshev loss maxj |σ(o)(j) − y(j)| hinge hinge [7] (margin) loss ∑ j max(0, 1 2 − ŷo) hinge(2) squared hinge (margin) loss ∑ j max(0, 1 2 − ŷo)(2) hinge(3) cu”
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild (2017, methodology): “In ICML 2013 competition [16], the winner [41] was based on Deep Convolutional Neural Network (DCNN) plus SVM.”
Applications of Support Vector Machine (SVM) Learning in Cancer Genomics. (2018, background): “SVM can be extended for multiclass problems using the so-called one-vs-rest approach (15).”
Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks (2019, methodology): “In this experiment, we replaced the squared difference term (p − q)2 in the DT-based loss function (Equation (8)) with some of the alternatives proposed in [33], [47], [48].”
Financial Time Series Forecasting – A Deep Learning Approach (2017, methodology): “Whilst still using the CNN layered architecture, [7] demonstrates the benefit of replacing the softmax activation function with a linear support vector machine.”
Going deeper in facial expression recognition using deep neural networks (2015, neutral): “[45] is the winner of the ICML 2013 Challenges on FER2013 database that employed a convolutional neural network similar to AlexNet [20] but with linear one-vs-all linear SVM top layer instead of a Softmax function.”
Simple convolutional neural network on image classification (2017, methodology): “Another commonly used method is SVM, which can be combined with CNNs to solve different classification tasks[18].”
Multitask Learning of Deep Neural Networks for Low-Resource Speech Recognition (2015, methodology): “Other ways such as weight pruning [54] and large-margin optimization [55], [56] have also been proposed, and it will be interesting to see if these methods are complementary to our proposed MTL methods.”