Deep Residual Networks with Exponential Linear Unit

By Anish Shah, Eashan Kadam, Hena Shah, Sameer Shinde. 2016

131 citations recorded by the source corpus.

The depth of convolutional neural networks is a crucial ingredient for reduction in test errors on benchmarks like ImageNet and COCO. However, training a neural network becomes difficult with increasing depth. Problems like vanishing gradient and diminishing feature reuse are quite trivial in very deep convolutional neural networks. The notable recent contributions towards solving these problems and simplifying the training of very deep models are Residual and Highway Networks. These networks allow earlier representations (from the input or those learned in earlier layers) to flow unimpededly to later layers through skip connections. Such very deep models with hundreds or more layers have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO. In this paper, we propose to replace the combination of ReLU and Batch Normalization with Exponential Linear Unit (ELU) in Residual Networks. Our experiments show that this not only speeds up the learning behavior in Residual Networks, but also improves the classification performance as the depth increases. Our model increases the accuracy on datasets like CIFAR-10 and CIFAR-100 by a significant margin.

Subfields: Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dan Hendrycks
2. Gang Zeng
3. Kevin Gimpel
4. Luc Van Gool
5. Falong Shen
6. Trevor Darrell
7. Alan L. Yuille
8. Dacheng Tao

In Industrial and Manufacturing Engineering

1. Dan Hendrycks

In Model Architectures

1. Kevin Gimpel
2. Dan Hendrycks
3. Gang Zeng
4. Ali Rahimi
5. Falong Shen
6. Ronghang Hu
7. Quoc V. Le
8. Francis R. Bach

Citations in the corpus

• Gaussian Error Linear Units (GELUs) (2016, background): “Some have noted that ELUs have an exploding gradient with residual networks (Shah et al., 2016), and this is alleviated with batch normalization at the end of a residual block.”
• Weighted residuals for very deep networks (2016, background): “The original residual networks stated above have two defects Incompatibility of ReLU and element-wise addition.”
• Convergence Results for Neural Networks via Electrodynamics (2017, background): “For example, better accuracy and training speed were reported when using the newly discovered exponential linear unit (ELU) activation function in [CUH15, SKS + 16].”