Searching for Activation Functions

By Prajit Ramachandran, Barret Zoph, Quoc V. Le. 2018

3,877 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “We also use swish activation (Ramachandran et al., 2018; Elfwing et al., 2018), ﬁxed Au-toAugment policy (Cubuk et al., 2019), and stochastic depth (Huang et al., 2016) with drop connect ratio 0.2.”
GLU Variants Improve Transformer (2020, methodology): “…[Raﬀel et al., 2019] 1 , we use a version with no bias: Subsequent work has proposed replacing the ReLU with other nonlinear activation functions such as Gaussian Error Linear Units, GELU( x ) = x Φ( x ) [Hendrycks and Gimpel, 2016], and Swish β ( x ) = xσ ( βx ) [Ramachandran et al., 2017].”
Attention Augmented Convolutional Networks (2019, background): “As such, it would be interesting to see if using Attention Augmentation as a primitive in automated architecture search procedures proves useful to find even better models than those previously found in image classification [55], object detection [12], image segmentation [6] and other domains[5, 1, ”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, methodology): “We also apply a regularization on the norm of scaling parameters in BN layers to ensure that a small mismatch in statistics is not ampliﬁed by BN. Swish Activation: The Swish activation [47], f ( u ) = u 1+ e − u , has been recently shown promising results in many applications [48, 49].”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “We use a linear layer with Swish activation [50] and batch-normalization as the projection block for these models.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “We also apply Swish activation [25] and dropout, which helps regularizing the network.”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “Moreover, we use the Swish activation function [23], which contributes a slight but consistent reduction in WER.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “• Transformer++: A Transformer with the following enhancements: RMS normalization [47], Swish activation [48] and a GLU multiplicative branch [38] in the feed forward inverted bottleneck (SwiGLU) [39].”
Designing Network Design Spaces (2020, methodology): “ReLU Many recent methods employ the Swish [22] activation function, e.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “The activation functions that we explored are: (1) GeLU (Hendrycks and Gimpel, 2016), (2) Swish (Ramachandran et al., 2017), (3) Exponential Linear Units (ELU) (Clevert et al., 2015), (4) Scaled exponential linear units (SeLU) (Klambauer et al., 2017), (5) Sigmoid and (6) Softplus.”
The Evolved Transformer (2019, methodology): “The best performing architecture search methods are those that are computationally intensive (Zoph & Le, 2017; Baker et al., 2016; Real et al., 2017; Xie & Yuille, 2017; Zoph et al., 2018; Real et al., 2019).”
Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, background): “The comparison of the nonlinear activations are studied in [44].”
E(n) Equivariant Graph Neural Networks (2021, methodology): “As non-linearity we used the Swish activation function (Ramachandran et al., 2017).”
Directional Message Passing for Molecular Graphs (2020, methodology): “First, DimeNet uses the self-gated Swish activation function σ(x) = x · sigmoid(x) (Ramachandran et al., 2018) instead of a regular ReLU activation function.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “The most popular and widely used activation functions include sigmoid, hyperbolic tangent (tanh), rectified linear units (ReLU) (Nair and Hinton, 2010), Maxout (Goodfellow et al., 2013), and Swish (Ramachandran et al., 2017a).”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Other specific architecture aspects have also been targeted, such as the hyperparameters [77, 50, 32, 44], activation functions [66], a specific layer [35], the full forward pass [23], the data augmentation [17, 62, 18], etc.”
AtomNAS: Fine-Grained End-to-End Neural Architecture Search (2019, methodology): “We ﬁrst pretrain AtomNAS models (without Swish activation function (Ramachandran et al., 2018) and Squeeze-and-Excitation (SE) module (Hu et al., 2018)) on ImageNet, use them as drop-in replacements for the backbone in the Mask-RCNN model (He et al., 2017a) by building the detection head on top of…”