Swish: a Self-Gated Activation Function

By Prajit Ramachandran, Barret Zoph, Quoc V. Le. 2017

712 citations recorded by the source corpus.

The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose a new activation function, named Swish, which is simply $f(x) = x \cdot \text{sigmoid}(x)$. Our experiments show that Swish tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

GLU Variants Improve Transformer (2020, methodology): “…[Raﬀel et al., 2019] 1 , we use a version with no bias: Subsequent work has proposed replacing the ReLU with other nonlinear activation functions such as Gaussian Error Linear Units, GELU( x ) = x Φ( x ) [Hendrycks and Gimpel, 2016], and Swish β ( x ) = xσ ( βx ) [Ramachandran et al., 2017].”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “Feature gating has been used for many tasks, such as machine translation [18], VQA [19], reinforcement learning [20], image classiﬁcation [21,22], and action recognition [23].”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “Moreover, meta-learning also encompasses the discovery of activation functions (Ramachandran et al., 2017) and optimizers (Bello et al.”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, background): “It is also worth noting that the simple architecture design makes it easy to equip ShuffeNets with the latest advances such as [13, 26].”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “The most popular and widely used activation functions include sigmoid, hyperbolic tangent (tanh), rectified linear units (ReLU) (Nair and Hinton, 2010), Maxout (Goodfellow et al., 2013), and Swish (Ramachandran et al., 2017a).”
Transfer Learning with Neural AutoML (2018, background): “At convergence, the cluster {Complaints, New Agg, Airline, Primary Emotion} is assigned (with high probability) a 1-layer networks with 256 units, Swish activation function, wide-layer learning rate 0.01, and dropout rate 0.2.”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, methodology): “Swish non-linearities (Ramachandran et al., 2017) and layer normalization (Ba et al., 2016) are used all across the network.”