Deep Learning using Rectified Linear Units (ReLU)

By Abien Fred Agarap. 2018

3,929 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, background): “In recent years, ReLU has become the most widely used activation function (Agarap, 2018).”
A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “The shape of input phone embeddings is (73, 384), and we use a ReLU (Agarap, 2018) nonlinear layer to transform 80-dim log-Mel ﬁlterbanks features to 384-dim.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, background): “Concretely, the similarity function s(vi, tj) = FC ( ReLU ( FC(Ũi[0, :]) )) , where FC is the linear projection, and the ReLU means ReLU activation function (Agarap, 2018).”
Collapse of Deep and Narrow Neural Nets (2018, background): “It has been known as one of the obstacles in training deep ReLU neural networks (Trottier et al., 2017; Agarap, 2018).”
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (2023, unknown): “Activation functions like ReLU [1] act as gates to selectively retain or discard values in a vector, inﬂuencing neuron activations in FC1 and FC2.”
Seamless retrievals of chlorophyll-a from Sentinel-2 (MSI) and Sentinel-3 (OLCI) in inland and coastal waters: A machine-learning approach (2020, background): “The final range scaling ensures all feature information is contained within the valid domain for the rectified linear unit (ReLU) activation functions (Agarap, 2018).”
Thin MobileNet: An Enhanced MobileNet Architecture (2019, methodology): “Both Batch normalization [11] and non–linear activation function ReLU [26] have been used after each convolution layer in the model.”
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, methodology): “We apply both 2-layer MLPs with ReLU activation [18] to map both audio and text outputs into 512 dimensions, which is the size of audio/text representations when training with the contrastive learning paradigm.”
An optimized dense convolutional neural network model for disease recognition and classification in corn leaf (2020, unknown): “Cifar 10 model was also optimized by adding more layers and using ReLU function.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, background): “This is followed by an activation function, typically ReLU or GELU [27,46].”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, background): “The update rule of hidden representation for GC is as follows where normalized adjacency matrix is learnable parameters of the l -th layer, and σ ( · ) indicates nonlinear activation function like ReLU [1].”
cosFormer: Rethinking Softmax in Attention (2022, methodology): “By comparing the results of φ ReLU with the softmax, we observe that models with softmax re-weighting converge faster and generalize better to downstream tasks.”
ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, methodology): “Then, taking the extracted features as inputs, we adopt two fully connected layers with ReLU nonlinearity [1] and batch normalization [5] to regress the IoU between RoIs and their corresponding ground-truths (or pseudo boxes) with sigmoid nonlinearity.”
How Artificial Intelligence Is Shaping Medical Imaging Technology: A Survey of Innovations and Applications (2023, background): “…unit (ReLU) [18] and its variants, which successfully addressed these issues and became integral components of modern deep learning architectures [19]. rectified linear unit (ReLU) [18] and its variants, which successfully addressed these issues and became integral components of modern deep…”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, unknown): “Each residual block consists of two convolution layers with kernel size 3, with instance normalization [58] and ReLU [5].”
ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, methodology): “It is composed of two deconvolution blocks, each of which contains one deconvolution layer followed by batch normalization [19] and ReLU [1].”