ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks

By Shuxuan Guo, José Manuel Álvarez, Mathieu Salzmann. 2018

92 citations recorded by the source corpus.

In this paper, we introduce an approach to training a given compact network. To this end, we leverage over-parameterization, which typically improves both optimization and generalization in neural network training, while being unnecessary at inference time. We propose to expand each linear layer, both fully-connected and convolutional, of the compact network into multiple linear layers, without adding any nonlinearity. As such, the resulting expanded network can benefit from over-parameterization during training but can be compressed back to the compact one algebraically at inference. We introduce several expansion strategies, together with an initialization scheme, and demonstrate the benefits of our ExpandNets on several tasks, including image classification, object detection, and semantic segmentation. As evidenced by our experiments, our approach outperforms both training the compact network from scratch and performing knowledge distillation from a teacher.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

Initialization and Regularization of Factorized Neural Layers (2021, methodology): “We also study a fledgling subfield we term overcomplete knowledge distillation (Arora et al., 2018; Guo et al., 2020; Cao et al., 2020) in which model weights are over-parameterized as overcomplete factorizations; after training the factors are multiplied to obtain a compact representation of the sa”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “Further improvements have been investigated for superior dynamic performance [19] and inference efficiency [5, 6, 9] recently.”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “In order to increase capacity of the these layers, we use linear train-time overparameterization as introduced in [13, 11, 12, 57, 18].”
Diverse Branch Block: Building a Convolution as an Inception-like Unit (2021, background): “ExpandNet [10], DO-Conv [1] and ACNet [8] can also be categorized into structural re-parameterization in the sense that they convert a block into a conv.”
RepVGG: Making VGG-style ConvNets Great Again (2021, background): “Asym Conv Block (ACB) [10], DO-Conv [1] and ExpandNet [11] can also be viewed as structural reparameterization in the sense that they convert a block into a conv.”