Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks

By Ameya Dilip Jagtap, Kenji Kawaguchi, G. Em Karniadakis. Proceedings of the Royal Society A, 2020

325 citations recorded by the source corpus.

We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each layer (layer-wise) and for every neuron (neuron-wise) separately, and then optimizing it using a variant of stochastic gradient descent algorithm. In order to further increase the training speed, an activation slope-based slope recovery term is added in the loss function, which further accelerates convergence, thereby reducing the training cost. On the theoretical side, we prove that in the proposed method, the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate, and that the gradient dynamics of the proposed method is not achievable by base methods with any (adaptive) learning rates. We further show that the adaptive activation methods accelerate the convergence by implicitly multiplying conditioning matrices to the gradient of the base method without any explicit computation of the conditioning matrix and the matrix–vector product. The different adaptive activation functions are shown to induce different implicit conditioning matrices. Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/7c6de94060d914b3d742d7fee24cdb2c6baa5710

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Physics-Informed Neural Operator for Learning Partial Differential Equations (2021, unknown): “We add a comparison experiment against the Locally adaptive activation functions for PINN (LAAF-PINN) [54] and Self-Adaptive PINN ( SA-PINN ) [55].”
Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations (2020, methodology): “Moreover, these activation slopes can also contribute to the loss function through the slope recovery term, see (8; 9) for more details.”
Parallel Physics-Informed Neural Networks via Domain Decomposition (2021, methodology): “Moreover, these activation slopes can also contribute to the loss function through the slope recovery term, see [26, 27] for more details.”
Physics-informed neural networks for inverse problems in supersonic flows (2022, methodology): “We employ the layer-wise adaptive activation function, denoted by σ ( na k ), where a k are the trainable parameters, n is the predeﬁned scaling factor and σ is the activation function, see Jagtap et al. [35, 36].”