On Feature Normalization and Data Augmentation

By Boyi Li, Felix Wu, Ser-Nam Lim, Serge J. Belongie, Kilian Q. Weinberger. 2020

164 citations recorded by the source corpus.

The moments (a.k.a., mean and standard deviation) of latent features are often removed as noise when training image recognition models, to increase stability and reduce training time. However, in the field of image generation, the moments play a much more central role. Studies have shown that the moments extracted from instance normalization and positional normalization can roughly capture style and shape information of an image. Instead of being discarded, these moments are instrumental to the generation process. In this paper we propose Moment Exchange, an implicit data augmentation method that encourages the model to utilize the moment information also for recognition models. Specifically, we replace the moments of the learned features of one training image by those of another, and also interpolate the target labels—forcing the model to extract training signal from the moments in addition to the normalized features. As our approach is fast, operates entirely in feature space, and mixes different signals than prior methods, one can effectively combine it with existing augmentation approaches. We demonstrate its efficacy across several recognition benchmark data sets where it improves the generalization capability of highly competitive baseline networks with remarkable consistency.

Subfields: Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “Image mixing is popular for classification [93], [94], [95], [96], [97], [98] yet, to the best of our knowledge, sees limited use for semantic segmentation [99], [100], [101], [102], [103], and has never been considered to design memory-efficient rehearsal learning systems.”
Image Data Augmentation for Deep Learning: A Survey (2022, unknown): “More recently, authors in [Li et al. , 2021] proposed an implicit data augmentation method, known as Moment Exchange, by encouraging the models to utilize the moment information of latent features.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “In other words, the style information is intuitively ‘editable’ by normalization [71], [243].”
Efficient Test-Time Model Adaptation without Forgetting (2022, background): “…(Wong et al., 2020; Rusak et al., 2020; Madaan et al., 2021), various data augmentation techniques (Lim et al., 2019; Hendrycks et al., 2020; Li et al., 2021; Hendrycks et al., 2021; Yao et al., 2022) and searching/enhancing sub-networks of a deep model (Zhang et al., 2021a; Guo et al.,…”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “…moments extracted from instance normalization and position normalization roughly capture the style and shape information of the image, Li et al. [281] proposed Moment Exchange, an implicit data augmentation method that encourages models to utilize moment information in recognition models as…”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “This operation relatively resembles PONO function [16] in the realm of CNNs.”
Towards Stable Test-Time Adaptation in Dynamic Wild World (2023, methodology): “…generalization (Shankar et al., 2018; Li et al., 2018a; Dou et al., 2019), increasing the training dataset size (Orhan, 2019), various data augmentation techniques (Lim et al., 2019; Hendrycks et al., 2020; Li et al., 2021; Hendrycks et al., 2021; Yao et al., 2022 Adaptation with Target Data.”
Shape-Texture Debiased Neural Network Training (2020, background): “…neural network training is related to a speciﬁc family of data augmentation, called Mixup (Zhang et al., 2018), which blends pairs of images and their labels in a convex manner, either at pixel-level (Zhang et al., 2018; Yun et al., 2019) or feature-level (Verma et al., 2019; Li et al., 2020).”
Continual Test-Time Domain Adaptation (2022, background): “A number of works have also shown that data augmentation [52] during training [14, 16, 32, 69] and during testing [1, 40, 73] can improve model robustness and generalizability.”
MEMO: Test Time Robustness via Adaptation and Augmentation (2021, methodology): “When used on top of a model trained with moment exchange and CutMix [20], MEMO achieves state-of-the-art performance among ResNet-50 models and single test point methods.”