Batch-shaping for learning conditional channel gated networks

By Babak Ehteshami Bejnordi, Tijmen Blankevoort, Max Welling. 2019

84 citations recorded by the source corpus.

We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gating the deep-learning architecture on a fine-grained-level. Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner. We also introduce a generally applicable tool $batch$-$shaping$ that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “a, conditional computation) methods have been developed in the image domain, achieving reduced computation by dynamically selecting channels [2, 25], skipping layers [42, 11, 40, 37], performing early exiting with auxiliary structures [22, 18, 3, 46], adaptively switching input resolutions [27, 36, ”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “GaterNet [6], dynamic channel pruned networks [16], batch-shaped channel-gated networks [2], and Conditional channel gated networks for continual learning [1] perform gating in a more fine-grained fashion by turning individual channels of a layer on or off conditioned on the input.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, unknown): “3.3.”
CAP: Context-Aware Pruning for Semantic Segmentation (2021, methodology): “Our baselines include BN-Scale, NS (Network Slimming [20]), FPGM [14] and the recently proposed conditional computing based method, CCGN [2].”
Dynamic Channel and Layer Gating in Convolutional Neural Networks (2020, methodology): “One would be to use the batch-shaping loss proposed in [2] for our model.”