Large Margin Deep Neural Networks: Theory and Algorithms
By Shizhao Sun, Liwei Wang, Tie-Yan Liu. 2015
22 citations recorded by the source corpus.
Deep neural networks (DNN) have achieved huge practical success in recent years. However, its theoretical properties (in particular generalization ability) are not yet very clear, since existing error bounds for neural networks cannot be directly used to explain the statistical behaviors of practically adopted DNN models (which are multi-class in their nature and may contain convolutional layers). To tackle the challenge, we derive a new margin bound for DNN in this paper, in which the expected 0-1 error of a DNN model is upper bounded by its empirical margin error plus a Rademacher Average based capacity term. This new bound is very general and is consistent with the empirical behaviors of DNN models observed in our experiments. According to the new bound, minimizing the empirical margin error can effectively improve the test performance of DNN. We therefore propose large margin DNN algorithms, which impose margin penalty terms to the cross entropy loss of DNN, so as to reduce the margin error during the training process. Experimental results show that the proposed algorithms can achieve significantly smaller empirical margin errors, as well as better test performances than the standard DNN algorithm.
Subfields: Artificial Intelligence, Industrial and Manufacturing Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Dilip Krishnan
- Hossein Mobahi
- Samy Bengio
- Taiji Suzuki
- Gamaleldin F. Elsayed
- Kevin Regan
- Hiroshi Abe
In Industrial and Manufacturing Engineering
Citations in the corpus
- Robust Large Margin Deep Neural Networks (2016, neutral): “Similar behaviour is also depicted in [34].”
- Large Margin Deep Networks for Classification (2018, background): “Prior work (Liu et al., 2016; Sun et al., 2015; Sokolic et al., 2016; Liang et al., 2017) has explored the benefits of encouraging large margin in the context of deep networks.”
- Spectral-Pruning: Compressing deep neural network via spectral analysis (2018, methodology): “Our analysis is categorized to so called fast learning rate that achieves O (1 /n ) convergence rate for sample size n unlike such theories as [44, 40, 7, 41] showing O (1 / √ n ) convergence.”