Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

By Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, Cho-Jui Hsieh. 2019

171 citations recorded by the source corpus.

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “Neural stochastic differential equations learn stochastic processes from observations along the path (Tzen & Ragin-sky, 2019; Jia & Benson, 2019; Liu et al., 2019; Li et al., 2020), but are not designed for marginal density estimation or asymptotically unbiased sampling.”
Neural Controlled Differential Equations for Irregular Time Series (2020, result): “[39, 40] investigate using stochasticity as a regularizer, and demonstrate better performance by doing so.”
TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed (2022, background): “Recent work has also extended neural ODEs to stochastic differential equations [62, 63], Riemannian manifold [64], Bayesian learning frameworks [65] and graph-structured data [66].”
Stochastic Normalizing Flows (2020, background): “2 Visualizing regularization As discussed in Liu et al. (2019), the stochastic noise injection in SDEs is a natural form of regularization, that can potentially improve robustness to noisy or adversarial data.”
GRAND: Graph Neural Diffusion (2021, background): “, 2020) and provided extensions into new domains such as stochastic (Liu et al., 2019) differential equations.”
TrajectoryNet: A Dynamic Optimal Transport Network for Modeling Cellular Dynamics (2020, unknown): “In future work, we would like to consider stochastic dynamics (Liu et al., 2019) and learning the growth term together with the dynamics.”
Scalable Gradients for Stochastic Differential Equations (2020, methodology): “Existing meth-ods in this regime [21, 43, 79] all require simulating a (forward) SDE where each step requires computing entire Jacobian matrices.”
Continuous-in-Depth Neural Networks (2020, background): “Other theoretical research fronts include interfacing Neural ODEs with normalizing flows [22, 79, 40, 56], graph NNs [61, 67], stochastic differential equations [52, 37, 24, 47] and RNNs [10, 65, 27, 18, 49].”