Adaptive activation functions accelerate convergence in deep and physics-informed neural networks

By Ameya Dilip Jagtap, G. Em Karniadakis. Journal of Computational Physics, 2019

1,081 citations recorded by the source corpus.

Abstract We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems. We theoretically prove that in the proposed method, gradient descent algorithms are not attracted to suboptimal critical points or local minima. Furthermore, the proposed adaptive activation functions are shown to accelerate the minimization process of the loss values in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion datasets with and without data augmentation.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/fdf7a8bf12b6ba625cd7662ea3cf4a48e39d32cb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs (2020, methodology): “One can possibly further reduce the error by using adaptive activation functions as suggested in [28].”
DeepXDE: A Deep Learning Library for Solving Differential Equations (2019, background): “We also note that acceleration of training can be achieved by using adaptive activation function that may remove bad local minima, see [24, 23].”
NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations (2020, methodology): “Such speed up can be further enhanced by using adaptive activation functions as demonstrated in [30, 31].”
Variational Physics-Informed Neural Networks For Solving Partial Differential Equations (2019, background): “The recent development of PINNs has been well established for forward and inverse problem of solving di ﬀ erential equations [30, 32, 39, 29] and since then a number of extensions has been made to tackle several physical and biomedical problems [28, 18, 19].”
Physics-informed neural networks (PINNs) for fluid mechanics: a review (2021, methodology): “We employ a 6×60 (6 hidden layer) neural network and train it by using layer-wise adaptive tanh activate function [35] and the Adam optimizer with the learning late being 6× 10−4”
Activation functions in deep learning: A comprehensive survey and benchmark (2021, methodology): “Recently, a trainable parameter was used in different non-adaptive AFs such as Sigmoid, Tanh, and ReLU to make it adaptive [90].”
Physics-informed neural networks for inverse problems in supersonic flows (2022, methodology): “We employ the layer-wise adaptive activation function, denoted by σ ( na k ), where a k are the trainable parameters, n is the predeﬁned scaling factor and σ is the activation function, see Jagtap et al. [35, 36].”
Physics‐Informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions (2021, methodology): “The interested reader is directed to Jagtap et al., 2020a&b for a discussion of adaptive activation functions and their effect on convergence.”
hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition (2020, background): “…employed in many physical problems such as discovering turbulence models from scattered / noisy measurements [28], high speed ﬂows [4], stochastic di ﬀ erential equation by generative adversarial networks [24], fractional di ﬀ erential equations [29], and adaptive activation functions [30, 31].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, methodology): “Euler equations are hyperbolic conservation laws that might permit discontinuous solutions such as shock and contact waves, and in particular a one dimensional Euler system is written as (Jagtap et al, 2020b) ∂U ∂t +∇ · f(U) = 0, x ∈ Ω ⊂ R2, where U = ρρu ρE f = ρu,p+ ρu2 pu+ ρuE given ρ as the ”