In-place Activated BatchNorm for Memory-Optimized Training of DNNs

By Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder. 2017

377 citations recorded by the source corpus.

In this work we present In-Place Activated Batch Normalization (INPLACE-ABN) - a novel approach to drastically reduce the training memory footprint of modern deep neural networks in a computationally efficient way. Our solution substitutes the conventionally used succession of BatchNorm + Activation layers with a single plugin layer, hence avoiding invasive framework surgery while providing straightforward applicability for existing deep learning frameworks. We obtain memory savings of up to 50% by dropping intermediate results and by recovering required information during the backward pass through the inversion of stored forward results, with only minor increase (0.8-2%) in computation time. Also, we demonstrate how frequently used checkpointing approaches can be made computationally as efficient as INPLACE-ABN. In our experiments on image classification, we demonstrate on-par results on ImageNet-1k with state-of-the-art approaches. On the memory-demanding task of semantic segmentation, we report competitive results for COCO-Stuff and set new state-of-the-art results for Cityscapes and Mapillary Vistas. Code can be found at https://github.com/mapillary/inplace_abn.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Meanwhile, we replace the standard Batchnorm with InPlace-ABN [1] to the mean and standard-deviation of BatchNorm across multiple GPUs.”
Panoptic Feature Pyramid Networks (2019, methodology): “To increase feature resolution, which is necessary for generating highquality results, recent top methods [12, 58, 5, 59] rely heavily on the use of dilated convolution [57] (also known as atrous convolution [10]).”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “First, it is a surprisingly expensive computational primitive, which incurs memory overhead (Rota Bulò et al., 2018), and significantly increases the time required to evaluate the gradient in some networks (Gitman & Ginsburg, 2017).”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, methodology): “9% on Cityscapes test set, and performs comparably with other ImageNet-pretrained state-of-the-art models [81, 87, 4, 11, 6] when also exploiting the coarse annotations on Cityscapes.”
ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation (2022, methodology): “In particular, SDR [13] and SSUL [3] use ResNet-101 provided by PyTorch [14], while MiB [2] and PLOP [6] exploit a variant of ResNet-101 using in-place ABN [1] layers.”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “The weights are initialized on ImageNet [65] and trained using momentum and weight decay proposed in [11].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “We also apply the in-place activated batch normalization [71] in the backbone pre-trained on the ImageNet [20], as the above methods.”
Modeling the Background for Incremental Learning in Semantic Segmentation (2020, methodology): “The backbone has been initialized using the ImageNet pretrained model [27].”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, methodology): “The previous academic competition elicited gains of 0.8% in mIOU from [97] to [6] over the span of one year.”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “We use INPLACE-ABNsync [47] to synchronize the mean and standard-deviation of BN across multiple GPUs.”
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (2019, background): “Bulo et al. [5] replace only ReLU and batch normalization layers with invertible variants, reconstructing their inputs during the backward pass, reducing memory usage up to 50%.”
PointRend: Image Segmentation As Rendering (2019, methodology): “0001 is applied; horizontal flipping, color augmentation [33], and crop bootstrapping [2] are used during training; scale traintime data augmentation resizes an input image from 0.”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, methodology): “In a final step, we detect pixels that belong to sky regions using a pre-trained semantic segmentation model [35] and set their disparity to the minimum disparity in the image.”
Learning to Fuse Things and Stuff (2018, background): “On one hand, the best semantic segmentation networks [3, 43, 6] focus on dense tensor-to-tensor classification architectures that excel at recognizing stuff categories like roads, buildings, or sky [1].”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, methodology): “We replace BatchNorm layers with synchronized Inplace-ABN [49], and adopt bilinear interpolation in all the upsampling operations.”
OCNet: Object Context Network for Scene Parsing (2018, methodology): “The batch size is 40 and we also employ the InPlaceABNSync.”