Augmented Neural ODEs

By Emilien Dupont, Arnaud Doucet, Yee Whye Teh. 2019

792 citations recorded by the source corpus.

We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “Such problems can be reduced by using mixtures of ﬂows (Dinh et al., 2019; Cornish et al., 2019) or augmenting the base space (Dupont et al., 2019) at the cost of limiting ex-pressibility or losing the exact density.”
Score-based Generative Modeling in Latent Space (2021, background): “However, previous works found that augmenting neural ODEs [16, 17] and more generally generative models [18, 19, 20, 21] with latent variables improves their expressivity.”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “VAEs also do not suffer from expressivity limitations that normalizing flows face (Dupont et al., 2019; Kong & Chaudhuri, 2020), and in fact, they have recently shown state-of-the-art generative results among non-autoregressive likelihood-based models (Vahdat & Kautz, 2020).”
Flow Matching for Generative Modeling (2022, methodology): “, using augmentation (Dupont et al., 2019), adding regularization terms (Yang & Karniadakis, 2019; Finlay et al.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Just as with the auxiliary-variable formulation of Dupont et al. (2019), we can think of the above as a mixture of flows, with z indexing each mixture component.”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, neutral): “The bottleneck problem is studied for discriminative invert-ible models including neural ODEs (Dupont et al., 2019) and i-RevNets (Jacobsen et al., 2018), where zeros are padded to the input data to increase the number of dimensions.”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “[27] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “ANFs lie at the 2019) also employs augmentation to improve the expressivity and stability of a neural ODE (Chen et al., 2018a), and they believe such a method can be used to reduce the cost of training a continuous normalizing ﬂow (Grathwohl et al., 2019).”
TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed (2022, background): “Augmented-NODE [29] and NANODE [50] used increased dimensionality to lift the restriction that the trajectories of ordinary differential equations cannot intersect.”
Multiscale Deep Equilibrium Models (2020, methodology): “Unlike prior implicit models, such as DEQs and Neural ODEs, an MDEQ solves for and backpropagates through synchronized equilibria of multiple feature representations at different resolutions.”
Stochastic Normalizing Flows (2020, background): “Doing all this involves some technical subtlety, and eﬀective neural ODE architectures remain the subject of ongoing research — see for example (Dupont et al., 2019; Gholami et al., 2019; Zhang et al., 2019).”
GRAND: Graph Neural Diffusion (2021, methodology): “Many follow-up works explored augmentation (Dupont et al., 2019) and regularization (Finlay et al., 2020) and provided extensions into new domains such as stochastic (Li et al., 2020a; Tzen & Raginsky, 2019) differential equations.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “Similar to augmented normalizing flows [94], this can be solved by providing additional dimensions for the flow to traverse [47].”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, methodology): “Theorem 3.1 shows we can avoid difﬁcult computations associated with J − 1 Θ in (14) ( i.e. solving an associated linear system/adjoint equation) in implicit network literature (Chen et al. 2018; Dupont, Doucet, and Teh 2019; Bai, Kolter, and Koltun 2019; Winston and Kolter 2020).”
Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows (2019, background): “To avoid this, Dupont et al. (2019) propose augmenting the data by appending auxiliary dimensions and learning a new mapping on this space.”
Pseudo Numerical Methods for Diffusion Models on Manifolds (2022, methodology): “…general applications, like certain neural ODEs (Chen et al., 2019; Dupont et al., 2019). improve the results of NCSNs also can be used in DDPMs. Additional, Song et al. (2020b) combine DDPMs and NCSNs under the framework of neural differential equations (Chen et al., 2019; Dupont et al., 2019).”
Liquid Time-constant Networks (2020, methodology): “A large number of alternative approaches have tried to improve and stabilize the adjoint method [18], use neural ODEs in speciﬁc contexts [47, 35] and to characterize them better [11, 12, 32, 21, 29, 45].”
Learning Differential Equations that are Easy to Solve (2020, background): “Dupont et al. (2019) provided theoretical arguments that adding extra dimensions to the state of a neural ODE should make training easier, and showed that this helped reduce NFE during training.”