Liquid Time-constant Networks

By Ramin M. Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu. AAAI Conference on Artificial Intelligence, 2020

462 citations recorded by the source corpus.

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics, and compute their expressive power by the trajectory length measure in a latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/1b9a07702cd346673b4c5e798d2256157fab1d3f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Christopher R'e
2. Soham De
3. Tri Dao
4. Albert Gu
5. Yoshua Bengio
6. Samuel L. Smith
7. Razvan Pascanu
8. Atri Rudra

In Statistical and Nonlinear Physics

1. Caglar Gulcehre
2. Christopher R'e
3. Albert Gu
4. Razvan Pascanu
5. Samuel L. Smith
6. Anushan Fernando
7. Soham De
8. Antonio Orvieto

Citations in the corpus

• Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “(non-diagonal) S4 formulation combined with liquid time-constant networks (Hasani et al., 2021; 2022).”
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, methodology): “Other models include the Liquid Time-Constant Networks (LTC) [27] and successor CfC [26], which use underlying dynamical systems with varying time-constants with stable behavior and provable rates of expressivity measured by trajectory length.”