Mish: A Self Regularized Non-Monotonic Activation Function

By Diganta Misra. British Machine Vision Conference, 2020

1,134 citations recorded by the source corpus.

We propose Mish , a novel self-regularized non-monotonic activation function which can be mathematically defined as: f ( x ) = x tanh ( softplus ( x )) . As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision (AP 50val ) by 2.1% in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by ≈ 1% while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8969d85db0abfc2e2739b752d38280745102eb26

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bernt Schiele
2. Yoshua Bengio
3. Quoc V. Le
4. Soham De
5. Matthias Hein
6. Razvan Pascanu
7. Mohit Bansal
8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Bernt Schiele
3. Dacheng Tao
4. Andrew Zisserman
5. Trevor Darrell
6. Xiaogang Wang
7. Yi Yang
8. Jitendra Malik

Citations in the corpus

• Resurrecting Recurrent Neural Networks for Long Sequences (2023, background): “, 2015), Hard Tanh and Mish (Misra, 2019) (ReLU and Tanh already in Table 6).”
• Relating Adversarially Robust Generalization to Flat Minima (2021, methodology): “, learning rate schedule, weight decay, batch size, or different activation functions [15, 39, 21], and methods explicitly improving flatness, e.”
• A review on modern defect detection models using DCNNs – Deep convolutional neural networks (2021, neutral): “…Cascade Mask R-CNN(ResNeXt152, multi-scale) [66] 53.3% 58.5% SpineNet-190 [67] 54.3 DetectoRS(ResNeXt-101-32x4d, multi-scale) [68] 54.7% 60.1% EfficientDet-D7x(multi-scale) [69] 55.1% 59.9% CSP-p6 + Mish(multi-scale) [70] 55.2% 60.7% DetectoRS(ResNeXt-101-64x4d, multi-scale) [68] 55.7% 61.1% ).”
• A Survey of Modern Deep Learning based Object Detection Models (2021, neutral): “network, including Mish activation [74], Cross-stage partial connections (CSP) [25], SPP-Block [41], PAN path aggregated block [46] , Multi input weighted residual connections (MiWRC), etc.”
• Slim-neck by GSConv: a lightweight-design for real-time detector architectures (2022, background): “Swish [38], Equation (3), and Mish [39], Equation (4), are generally batter than ReLU [40] for training DNNs because of smooth.”