Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

By Wenling Shang, Kihyuk Sohn, Diogo Almeida, Honglak Lee. 2016

530 citations recorded by the source corpus.

Recently, convolutional neural networks (CNNs) have been used as a powerful tool to solve many problems of machine learning and computer vision. In this paper, we aim to provide insight on the property of convolutional neural networks, as well as a generic method to improve the performance of many CNN architectures. Specifically, we first examine existing CNN models and observe an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). Inspired by our observation, we propose a novel, simple yet effective activation scheme called concatenated ReLU (CRelu) and theoretically analyze its reconstruction property in CNNs. We integrate CRelu into several state-of-the-art CNN architectures and demonstrate improvement in their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer trainable parameters. Our results suggest that better understanding of the properties of CNNs can lead to significant performance improvement with a simple modification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

GAN Inversion: A Survey (2021, background): “Lei et al. [107] consider a one-layer model in the form of G ¼ gðzÞ ¼ ReLUðWzþ bÞ.”
Large Scale Adversarial Representation Learning (2019, methodology): “Given the global average pooling output a, we first compute h = BatchNorm(a), and the final feature is computed by concatenating [ReLU(h),ReLU(−h)], sometimes called a “CReLU” (concatened ReLU) non-linearity [29].”
The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, methodology): “Firstly, as was shown by (Shang et al., 2016), convolutional neural networks with rectified linear units tend to lead to pairs of opposite features.”
Improving GANs Using Optimal Transport (2018, methodology): “The main module of the critic is a convolution with a 5 × 5 kernel and stride 2 using the concatenated ReLU activation function (Shang et al., 2016).”
Swish: a Self-Gated Activation Function (2017, background): “One-to-many functions, like Concatenated ReLU (Shang et al., 2016), improve performance by extracting more nonlinear signal from a single source.”
Steerable CNNs (2016, positive): “If ρ is realized by a signed permutation matrix, then CReLU(α) = (ReLU(α),ReLU(−α)) introduced by Shang et al. (2016), or any concatenated nonlinearity ν′(α) = (ν(α), ν(−α)), will be admissible.”
Sorting out Lipschitz function approximation (2018, background): “For example, ReLU, Leaky ReLU, concatenated ReLU (Shang et al., 2016), and Maxout (Goodfellow et al.”
MLP-Mixer: An all-MLP Architecture for Vision (2021, neutral): “Similar to CNNs, we observe many pairs of feature detectors with opposite phases [39].”
The Shattered Gradients Problem: If resnets are the answer, then what is the question? (2017, positive): “We use concatenated rectifiers or CReLUs (Shang et al., 2016): x 7→ ( ρ(x) ρ(−x) ) The key observation is that initializing weights with a mirrored block structure yields linear outputs ( W −W ) · ( ρ(x) ρ(−x) ) = Wρ(x)−Wρ(−x) = Wx.”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, methodology): “Every layer, except the LSTM, was followed by a concatenated ReLU layer [89] effectively doubling the number of outputs to 16 for following layers.”
Doubly Convolutional Neural Networks (2016, background): “[10] observes that filters learned by ReLU CNNs often contain pairs with opposite phases in the lower layers.”
Learning towards Minimum Hyperspherical Energy (2018, background): “, weights of kernels/filters in convolutional neural networks (CNNs)) caused by over-parametrization presents an issue [37, 41], which motivated a series of influential works in network compression [10, 1] and parameter-efficient network architectures [16, 19, 62].”
Speaker-independent raw waveform model for glottal excitation (2018, background): “This is followed by two 1 × 1 convolution layers with contenated rectifier activations [15], whose output is finally projected to the mixture density network output of size 3K (where K is the number of mixture components).”
Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion (2021, background): “Unless otherwise speciﬁed, we consider the widely used ReLU( · ) = max(0 , · ) as the activation function.”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “For example, the work in [44] found that the lower convolution layers of CNNs learned redundant filters to extract both positive and negative phase information of an input signal, and defined T (·) to be the simple negation function:”
Recent Advances in Deep Learning for Object Detection (2019, background): “CReLu [172] layer to reduce non-linear computation and acceler1473”
Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation (2017, background): “Many promising techniques have been developed, such as novel network structures [29, 13, 41], non-linear activation functions [12, 7, 6, 38], pooling strategies [11, 8, 53] and objective loss functions [43, 36], etc.”
PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection (2016, unknown): “ReLU) [2] is applied to the early stage of our CNNs (i.”