Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

By Stefan Elfwing, Eiji Uchibe, Kenji Doya. 2017

2,602 citations recorded by the source corpus.

In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10 × 10 board, using TD(λ) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa(λ) agent with SiLU and dSiLU hidden units.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “We also use swish activation (Ramachandran et al., 2018; Elfwing et al., 2018), ﬁxed Au-toAugment policy (Cubuk et al., 2019), and stochastic depth (Huang et al., 2016) with drop connect ratio 0.2.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “Feature gating has been used for many tasks, such as machine translation [18], VQA [19], reinforcement learning [20], image classiﬁcation [21,22], and action recognition [23].”
The Evolved Transformer (2019, methodology): “The activation vocabulary is { SWISH (Ramachandran et al., 2017; Elfwing et al., 2018), RELU , LEAKY RELU (M AAS ET AL ., 2013), NONE } .”
Searching for MobileNetV3 (2019, methodology): “In [36, 13, 16] a nonlinearity called swish was introduced that when used as a drop-in replacement for ReLU, that significantly improves the accuracy of neural networks.”
Diff-TTS: A Denoising Diffusion Model for Text-to-Speech (2021, methodology): “The decoder network consists of a stack of 12 residual blocks with Conv1D, tanh, sigmoid, and 1x1 convolutions with 512 residual channel [29].”
Bottleneck Transformers for Visual Recognition (2021, background): “Other aspects of improved training of backbone architectures for image classification has been the use of SqueezeExcitation (SE) blocks [9], SiLU non-linearity [5, 12, 8] and further lowering the weight decay (for eg.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, methodology): “We focus on 4 different components based on the design of EfficientNet [51], 2 architecture modules and 2 regularizations: Squeeze-and-Excitation (SE) [20], a channel attention module used after the spatial convolution; SiLU/Swish-1 [40, 11, 17], an activation function with the form x·sigmoid(x); Ra”
Best Practices for Scientific Research on Neural Architecture Search (2019, methodology): “For example, for image datasets, next to the choice of optimizer and number of training epochs, important choices include activation functions (e.g., Swish (Elfwing et al., 2018)), learning rate schedules (e.g., cosine annealing (Loshchilov & Hutter, 2017)), data augmentation (e.g. by CutOut (Devrie”
MixConv: Mixed Depthwise Convolutional Kernels (2019, neutral): “Our search space also includes swish activation [4, 20], squeeze-and-excitation module [8] (similar to [29]), and grouped convolutions with group size 1 or 2 for 1x1 convolutions (similar to [30]).”
Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis (2022, methodology): “Therefore, in our proposed CFM, there are two types of modulation conducted upon normalized features sequentially, between which an extra convolution (conv) and SiLU layer (Elfwing, Uchibe, and Doya 2018) are inserted.”
ASF-YOLO: A Novel YOLO Model with Attentional Scale Sequence Fusion for Cell Instance Segmentation (2023, methodology): “• Finally, 3D convolution, 3D batch normalization, and SiLU [46] activation function are used to complete scale sequence feature extraction.”
MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model (2022, positive): “The residual block receives the time embedding through a linear layer, SiLU activation, and another linear layer.”
MagicVideo: Efficient Video Generation With Latent Diffusion Models (2022, neutral): “SiLU denotes the sigmoid ReLU [8].”
Parallelizing Linear Transformers with the Delta Rule over Sequence Length (2024, positive): “We instead use the SiLU activation [23], which was found to perform better [86, 18].”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, methodology): “The output of the sigmoid function is multiplied with its input in sigmoid-weighted linear unit (SiLU) AF [23] as”
Autoregressive Image Generation without Vector Quantization (2024, methodology): “Each block sequentially applies a LayerNorm (LN) [1], a linear layer, SiLU [12], and another linear layer, merging with a residual connection.”
A survey on modern trainable activation functions (2020, background): “it is worth pointing out that for α→ +∞, Swish behaves as ReLU, while, if α = 1, it becomes equal to SiLU (Elfwing et al., 2018).”