A Proposal on Machine Learning via Dynamical Systems

By W. E. Communications in Mathematics and Statistics, 2017

897 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/e3f8c8253767b6fe0891025cdf772cd286337921

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Reversible Architectures for Arbitrarily Deep Residual Neural Networks (2017, background): “Deep learning powers many research areas and impacts various aspects of society (LeCun, Bengio, and Hinton 2015) from computer vision (He et al. 2016; Huang et al. 2017), natural language processing (Cho et al. 2014) to biology (Esteva et al. 2017) and e-commerce.”
Functional Gradient Boosting based on Residual Network Perception (2018, background): “Moreover, several studies (Weinan, 2017; Haber et al., 2017; Chang et al., 2017a;b; Lu et al., 2017) have pointed out that ResNet layers can be regarded as discretization steps of ordinary differential equations.”
Neural Controlled Differential Equations for Irregular Time Series (2020, background): “Indeed the connection between RNNs and dynamical systems is well-known [1, 2, 3, 4].”
Deep Neural Networks Motivated by Partial Differential Equations (2018, methodology): “We derived a continuous interpretation of the filtering provided by ResNets in [17] and similar observations were made in [13, 8].”
Model Reduction and Neural Networks for Parametric PDEs (2020, methodology): “…examines the forward propagation of neural networks as the ﬂow of a time-dependent PDE, combining the continuous time formulation of ResNet [34, 80] with the idea of neural networks acting on spaces of functions: by considering the initial condition as a function, this ﬂow map may be thought…”
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise (2019, positive): “The idea of formulating ResNet as a dynamic system was discussed in [8].”
Camouflaged Object Detection with Feature Decomposition and Edge Reconstruction (2023, methodology): “Compared with the traditional residual network structure that can be seen as the first-order Euler discretization approximation of ODE with nonnegligible truncation errors [46], the proposed OER module employs a higherorder ODE solver, specifically, a second-order Runge-Kutta (RK2), to provide more ”
Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers (2020, background): “Other studies have investigated the similarities of dynamical systems and deep learning methods [65] and employed conservation laws to learn systems described by Hamiltonian mechanics [18, 12].”
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations (2018, result): “This connection between dynamical systems and feed-forward architectures was recently also observed by several other authors [48].”
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, methodology): “An interesting fact is that some popular networks in computer vision, such as ResNet[15, 16], have close relationship with ODEs/PDEs and can be naturally merged with traditional computational mathematics in various tasks [17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View (2019, methodology): “Inspired by the relationship between the ODE and neural networks [41, 21, 15, 6, 44, 33, 37], we show that the Transformer layers can be naturally interpreted as a numerical ODE solver for a first-order convectiondiffusion equation in a multi-particle dynamic system (MPDS).”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, background): “Neural ODEs leverage known connections between deep residual models and discretizations of differential equations (Haber and Ruthotto 2017; Weinan 2017; Ruthotto and Haber 2019; Chang et al. 2018; Finlay et al. 2020; Lu et al. 2018), and replace these discretizations by black-box ODE solvers in forw”
Augmented Neural ODEs (2019, neutral): “The relationship between neural networks and differential equations has been studied in several recent works (Weinan, 2017; Lu et al., 2017; Haber & Ruthotto, 2017; Ruthotto & Haber, 2018; Chen et al., 2018).”