MixConv: Mixed Depthwise Convolutional Kernels

By Mingxing Tan, Quoc V. Le. 2019

471 citations recorded by the source corpus.

Depthwise convolution is becoming increasingly popular in modern efficient ConvNets, but its kernel size is often overlooked. In this paper, we systematically study the impact of different kernel sizes, and observe that combining the benefits of multiple kernel sizes can lead to better accuracy and efficiency. Based on this observation, we propose a new mixed depthwise convolution (MixConv), which naturally mixes up multiple kernel sizes in a single convolution. As a simple drop-in replacement of vanilla depthwise convolution, our MixConv improves the accuracy and efficiency for existing MobileNets on both ImageNet classification and COCO object detection. To demonstrate the effectiveness of MixConv, we integrate it into AutoML search space and develop a new family of models, named as MixNets, which outperform previous mobile models including MobileNetV2 [20] (ImageNet top-1 accuracy +4.2%), ShuffleNetV2 [16] (+3.5%), MnasNet [26] (+1.3%), ProxylessNAS [2] (+2.2%), and FBNet [27] (+2.0%). In particular, our MixNet-L achieves a new state-of-the-art 78.9% ImageNet top-1 accuracy under typical mobile settings (<600M FLOPS). Code is at this https URL tensorflow/tpu/tree/master/models/official/mnasnet/mixnet

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Dacheng Tao
2. Wanli Ouyang
3. Luc Van Gool
4. Yi Yang
5. Jiashi Feng
6. Chunhua Shen
7. Xiaogang Wang
8. Wenguan Wang

Citations in the corpus

• AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “For example, by allowing g to be a combination of convolutions with different kernel sizes, the MixConv block in MixNet (Tan & Le, 2019b) becomes a special case in our search space.”
• FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (2019, methodology): “Even without mixed kernels [44], FairNAS-B (77.”
• EfficientNetV2: Smaller Models and Faster Training (2021, methodology): “On the other hand, we reduce the search space size by (1) removing unnecessary search options such as pooling skip ops, since they are never used in the original Efficient-Nets; (2) reusing the same channel sizes from the backbone as they are already searched in (Tan & Le, 2019a).”
• DeepViT: Towards Deeper Vision Transformer (2021, methodology): “Motivated by the success of deep CNNs [9, 30, 35, 36], we conduct systematic study in the changes of the performance of ViTs as depth increases.”
• YOLOv4: Optimal Speed and Accuracy of Object Detection (2020, methodology): “• For VPU - we use grouped-convolution, but we refrain from using Squeeze-and-excitement (SE) blocks - specifically this includes the following models: EfficientNet-lite / MixNet [76] / GhostNet [21] / MobileNetV3”
• Neural Architecture Search for Lightweight Non-Local Networks (2020, background): “Using MobileNetV2 basic blocks in search space [3, 40, 33, 35, 31, 14, 26] significantly improves the performance of searched architectures.”
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, methodology): “Tables 2 and 3 show the performance impact of quantization with all the configurations of the proposed method on EfficientNet-B0 [26] and MixNet-S [27], respectively.”
• Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning (2020, background): “In addition, recent works have proposed new types of layers that consist of smaller operators arranged in parallel, such as MixConv [36] and Split-Attention [39] blocks.”
• Lite-HRNet: A Lightweight High-Resolution Network (2021, methodology): “MixNet [39] applies mixed kernels on the depthwise convolutions.”
• Revisiting Dynamic Convolution via Matrix Decomposition (2021, background): “Tan & Le (2019b) mix up multiple kernel sizes in a single convolution.”
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (2019, unknown): “1 MixNet-M [35] 360 5.”
• CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention (2021, methodology): “Since GoogLeNet (Szegedy et al., 2015) and MixConv (Tan & Le, 2019) use multi-scale convolutional kernels at every layer of models, the largest kernel size can only be 7 × 7 , and larger kernel size will induce unaccpetable computational budget.”
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search (2020, unknown): “2 MixNet-M† [34] 360 31.”
• SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search (2019, result): “In particular, our smallest model SCARLET-C is close to MixNet-M, saving 22% FLOPS.”
• Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “These methods have paved the way for the development of widely used lightweight CNNs, such as MobileNets [20, 21, 44], ShuffleNets [35, 65], MixNet [49], MNASNet [47] and RepVGG [12].”
• Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation (2019, unknown): “Model CIFAR-10 Acc CIFAR-100 Acc MixNet-M[31] 97.”
• Zero-Cost Proxies for Lightweight NAS (2021, methodology): “PytorchCV contains common state-of-the-art neural networks such as ResNets (He et al., 2016), DenseNets (Huang et al., 2017), MobileNets (Howard et al., 2017) and EfficientNets (Tan & Le, 2019a) – a representative assortment of top-performing models.”