Normalization

By Frank Schimmelfennig, Thomas Winzen. 2020

Focusing on the 2004 and 2007 Eastern enlargement of the European Union, this chapter traces the ‘normalization’ of differentiated integration after new countries become member states. The chapter explains pre-accession tensions between demands of member states to limit the membership benefits of the applicant countries, and demands of the applicants to help them deal with the burdens of joining a competitive European market. It also explores the post-accession process in which new member states are in a better position to avoid discriminatory differentiation but still have to cope with the repercussions of accession differentiation. The empirical analysis shows that almost all accession differentiation—both preferential and discriminatory—disappears within ten to fifteen years of membership.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “In the scale case, the first deconvolution is followed by LayerNorm (LN) [1] and GeLU [28].”
Adversarial Video Generation on Complex Datasets (2019, methodology): “We further tried Layer Normalization (Lei Ba et al., 2016), Instance Normalization (Ulyanov et al., 2016), and no normalization, but found that these signiﬁcantly underperformed Batch Normalization.”
Sub-Linear Memory: How to Make Performers SLiM (2020, background): “…of attention heads ( V ∈ R d model × d are trainable parameters (separate for each instance of MultiHead - Att , FFN ), “ + ” is broadcasted rowwise when biases are added and LN is layer normalization (Ba et al., 2016), which is applied row-wise and depends on additional trainable parameters.”
Intriguing properties of adversarial training (2019, methodology): “Different methods are proposed to exploit batch-wise ( e.g. , BN (Ioffe & Szegedy, 2015)), layer-wise ( e.g. , layer normalization (Ba et al., 2016)) or channel-wise ( e.g. , instance normalization (Ulyanov et al., 2016) and group normalization (Wu & He, 2018)) information for estimating…”
Smooth Adversarial Training (2020, background): “We conjecture this improvement is possibly due to larger image resolution (1) enables attackers to create stronger adversarial examples [17] (which will hurt accuracy but improve robustness); and (2) increases network capacity for better representation learning [64] (which will improve both…”
MOMENT: A Family of Open Time-series Foundation Models (2024, methodology): “Specifically, we remove the additive bias from the Layer Norm (Ba et al., 2016), and place it before the residual skip connections (He et al., 2016), and use the relation positional embedding scheme (Shaw et al., 2018).”
NormFormer: Improved Transformer Pretraining with Extra Normalization (2021, methodology): “The original transformer architecture (Vaswani et al., 2017) applies Layer Normalization (Ba et al., 2016) after each sublayer’s residual connection (“Post-LN”) in order to reduce the variance of the inputs to the following sublayer, i.e.: PostLN( x ) = LayerNorm( x + Sublayer( x )) , with where γ…”
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021, background): “In addition, after each sub-layer, there is a residual connection (He et al., 2015; Wang et al., 2019) followed by a layer normalization (LN) (Ba et al., 2016).”
DreamFusion: Text-to-3D using 2D Diffusion (2022, methodology): “Our NeRF MLP consists of 5 ResNet blocks (He et al., 2016) with 128 hidden units, Swish/SiLU activation (Hendrycks & Gimpel, 2016), and layer normalization (Ba et al., 2016) between blocks.”
Differentiable Learning-to-Normalize via Switchable Normalization (2018, methodology): “In recent years, many normalization methods such as Batch Normalization (BN) (Ioffe & Szegedy, 2015), Instance Nor-malization (IN) (Ulyanov et al., 2016), and Layer Normalization (LN) (Ba et al., 2016) have been developed.”