Skip Connections Eliminate Singularities

By Emin Orhan, Xaq Pitkow. 2017

257 citations recorded by the source corpus.

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Optimization & Training

Citations in the corpus

Similarity of Neural Network Representations Revisited (2019, background): “Invariance to orthogonal transformation implies invariance to permutation, which is needed to accommodate symmetries of neural networks (Chen et al., 1993; Orhan & Pitkow, 2018).”
Gradient Descent with Identity Initialization Efficiently Learns Positive-Definite Linear Transformations by Deep Residual Networks (2018, background): “…treatments of neural network learning algorithms include Lee et al. (1996); Arora et al. (2014); Livni et al. (2014); Janzamin et al. (2015); Safran & Shamir (2016); Zhang et al. (2016); Nguyen & Hein (2017); Zhang et al. (2017); Orhan & Pitkow (2018), although these are less closely related.”
Extreme Network Compression via Filter Group Approximation (2018, background): “The problems of training instability are lightened in ResNet due to the existence of Batch Normalization [15] and short-cut connection [26].”
Information-Theoretic Local Minima Characterization and Regularization (2019, methodology): “As well demonstrated in Orhan & Pitkow (2018), network with skip connection, e.g. ResNet (He et al., 2016a), WRN (Zagoruyko & Komodakis, 2016), and DenseNet (Huang et al., 2017) used in our experiments, can effectively eliminate all the aforementioned singularity.”
Lightlike Neuromanifolds, Occam's Razor and Deep Learning (2019, background): “It was shown [28] that skip connections are helpful to reduce the effect of singularities.”
Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis (2020, unknown): “Orhan and Pitkow [2017] shows that skip connections eliminate the singularity.”
Automatically Designing CNN Architectures Using the Genetic Algorithm for Image Classification (2018, background): “For example, the skip connections have also been claimed to be able to eliminate singularities [38].”
Degeneration in VAE: in the Light of Fisher Information Loss (2018, background): “They can skip one or more layers of nonlinear mapping and have shown promise in typical deep neural network [He et al., 2016; Orhan and Pitkow, 2017].”
Completely Automated CNN Architecture Design Based on Blocks (2020, background): “Efforts in [37] and [38] have been put on investigating the mechanism behind the success of RBs and DBs and revealed that RBs and DBs are able to mitigate the adverse impact of the gradient vanishing problem [39], based on which a deep architecture is capable of effectively learning the…”
Automatically Designing CNN Architectures Using Genetic Algorithm for Image Classification (2018, background): “For example, the skip connections have also been claimed to be able to eliminate singularities [36].”
CNN-Based Projected Gradient Descent for Consistent CT Image Reconstruction (2017, background): “The intrinsic skip connections help to eliminate singularities during the training [50].”