Positional Normalization

By Boyi Li, Felix Wu, Kilian Q. Weinberger, Serge J. Belongie. 2019

110 citations recorded by the source corpus.

A popular method to reduce the training time of deep neural networks is to normalize activations at each layer. Although various normalization schemes have been proposed, they all follow a common theme: normalize across spatial dimensions and discard the extracted statistics. In this paper, we propose an alternative normalization method that noticeably departs from this convention and normalizes exclusively across channels. We argue that the channel dimension is naturally appealing as it allows us to extract the first and second moments of features extracted at a particular image position. These moments capture structural information about the input image and extracted features, which opens a new avenue along which a network can benefit from feature normalization: Instead of disregarding the normalization constants, we propose to re-inject them into later layers to preserve or transfer structural information in generative networks. Codes are available at https://github.com/Boyiliee/PONO.

Subfields: Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Language-driven Semantic Segmentation (2022, methodology): “…property: a simple cascade of depthwise convolutions (Chollet, 2017) followed by non-linear activations (DepthwiseBlock), and another block that additionally augments the depthwise convolutions with the result of a max-pooling operation over the set of labels (BottleneckBlock) (Li et al., 2019).”
High-Fidelity Generative Image Compression (2020, background): “Most closely related, Positional Normalization [27] also normalizes over channels, but does not use learned αc, βc.”
Rethinking Spatial Dimensions of Vision Transformers (2021, background): “Therefore, layer normalization of ViT is closer to positional normalization [22] than a layer norm of convolutional neural network [1, 39].”
On Feature Normalization and Data Augmentation (2020, methodology): “We study how MoEx performs when using moments from LayerNorm (LN) [1], InstanceNorm (IN) [55], PONO [34], GroupNorm (GN) [60], and local response normalization (LRN) [32] perform.”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, background): “As opposed to [38] which computes layer-wise statistics for batch normalization (BN), we empirically find the normalization that computes the statistics at each spatial position, the positional normalization (PN) [26], better preserves the structure information synthesized in prior layers.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “This works like PN [71] but only uses the scaling operation.”