Mish: A Self Regularized Non-Monotonic Neural Activation Function

By Diganta Misra. arXiv.org, 2019

753 citations recorded by the source corpus.

The concept of non-linearity in a Neural Network is introduced by an activation function which serves an integral role in the training and performance evaluation of the network. Over the years of theoretical research, many activation functions have been proposed, however, only a few are widely used in mostly all applications which include ReLU (Rectified Linear Unit), TanH (Tan Hyperbolic), Sigmoid, Leaky ReLU and Swish. In this work, a novel neural activation function called as Mish is proposed. The experiments show that Mish tends to work better than both ReLU and Swish along with other standard activation functions in many deep networks across challenging datasets. For instance, in Squeeze Excite Net- 18 for CIFAR 100 classification, the network with Mish had an increase in Top-1 test accuracy by 0.494% and 1.671% as compared to the same network with Swish and ReLU respectively. The similarity to Swish along with providing a boost in performance and its simplicity in implementation makes it easier for researchers and developers to use Mish in their Neural Network Models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/3a5f83be8f7c1ba567d0ca4ed4b1c1e6838f8be5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive Review of YOLO: From YOLOv1 to YOLOv8 and Beyond (2023, methodology): “The best-performing architecture was a modiﬁcation of Darknet-53 with cross-stage partial connections (CSPNet) [61], and Mish activation function [57] as the backbone (see Figure 11.”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “All components of TD-MPC 2 are implemented as MLPs with intermediate linear layers followed by LayerNorm (Ba et al., 2016) and Mish (Misra, 2019) activations.”
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models (2021, neutral): “First, xt is transformed to hidden through a 2Dconvolution block which consists of one 2D-convolutional layer and Mish activation [Misra, 2019].”
A Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS (2023, methodology): “The best-performing architecture was a modification of Darknet-53 with cross-stage partial connections (CSPNet) [69], and Mish activation function [65] as the backbone (see Figure 12.”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, methodology): “Each block consisted of two temporal convolutions, each followed by group norm (Wu & He, 2018), and a final Mish nonlinearity (Misra, 2019).”
Classification of Hyperspectral Image Based on Double-Branch Dual-Attention Mechanism Network (2019, methodology): “The derivative of Mish is defined as: 𝑓′(𝑥) = 𝑒𝑥𝜔 𝛿2 (6) where 𝜔 = 4(𝑥 + 1) + 4𝑒𝑥 + 𝑒3𝑥 + 𝑒𝑥(4𝑥 + 6) and 𝛿 = 2𝑒𝑥 + 𝑒2𝑥 + 2.”
YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications (2022, background): “Combinations of convolutional layers and activation functions YOLO series adopted a wide range of activation functions, ReLU [27], LReLU [25], Swish [31], SiLU [4], Mish [26] and so on.”
Smooth Adversarial Training (2020, background): “We believe these results can be furthered if more advanced smooth alternatives ( e.g ., [40, 35, 5]) are used.”
A survey of the recent architectures of deep convolutional neural networks (2019, methodology): “One of the recently proposed activation function is MISH, which has shown better performance than ReLU in most of the recently proposed deep networks on benchmark datasets (Misra 2019).”
YOLOv4: Optimal Speed and Accuracy of Object Detection (2020, methodology): “Subsequently, LReLU [54], PReLU [24], ReLU6 [28], Scaled Exponential Linear Unit (SELU) [35], Swish [59], hard-Swish [27], and Mish [55], etc.”
Dynamic ReLU (2020, neutral): “A number of smooth activation functions have been developed to address this, such as softplus [6], ELU [4], SELU [19], Mish [26].”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, methodology): “[27], SELU [52], GELU [101], CELU [53], Softplus [93], Swish [29], ABReLU [44], LiSHT [24], Soft-Root-Sign (SRS) [26], Mish [99], PAU [111] and PDELU [59].”
Learning to Embed Categorical Features without Embedding Tables for Recommendation (2020, methodology): “Table 5 shows the results of with and without BN and Mish.”
Planning with Diffusion for Flexible Behavior Synthesis (2022, methodology): “Each block consisted of two temporal convolutions, each followed by group norm (Wu & He, 2018), and a ﬁnal Mish nonlinearity (Misra, 2019).”