Rethinking Skip Connection with Layer Normalization

By Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, Yuexian Zou. 2020

85 citations recorded by the source corpus.

Skip connection is a widely-used technique to improve the performance and the convergence of deep neural networks, which is believed to relieve the difficulty in optimization due to non-linearity by propagating a linear component through the neural network layers. However, from another point of view, it can also be seen as a modulating mechanism between the input and the output, with the input scaled by a pre-defined value one. In this work, we investigate how the scale factors in the effectiveness of the skip connection and reveal that a trivial adjustment of the scale will lead to spurious gradient exploding or vanishing in line with the deepness of the models, which could by addressed by normalization, in particular, layer normalization, which induces consistent improvements over the plain skip connection. Inspired by the findings, we further propose to adaptively adjust the scale of the input by recursively applying skip connection with layer normalization, which promotes the performance substantially and generalizes well across diverse tasks including both machine translation and image classification datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Wei Wang
6. Luo Si
7. Haiyang Xu
8. Ji Zhang

In Computer Vision and Pattern Recognition

1. Chenliang Li
2. Songfang Huang
3. Bin Bi
4. Ming Yan
5. Wei Wang
6. Haiyang Xu
7. Luo Si
8. Ji Zhang

In Model Architectures

1. Chenliang Li
2. Haiyang Xu
3. Jingren Zhou
4. Ming Yan
5. Luo Si
6. Xueguang Yuan
7. Dandan Li
8. Xiaohong Huang

In Optimization & Training

1. Chenliang Li
2. Haiyang Xu
3. Jingren Zhou
4. Ming Yan
5. Luo Si
6. Guohai Xu
7. Ji Zhang
8. Songfang Huang

Citations in the corpus

• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022, background): “A variety of skip connection methods have been proposed in recent years [22, 21, 12, 23, 24, 25].”
• MISSFormer: An Effective Transformer for 2D Medical Image Segmentation (2022, methodology): “Block: Inspired by the above exploration and [37], we extend the redesigned FFN of MISSFormer_S to make it more general.”