HiPPO: Recurrent Memory with Optimal Polynomial Projections

By Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher R'e. 2020

1,021 citations recorded by the source corpus.

A central problem in learning from sequential data is representing cumulative history in an incremental fashion as more data is processed. We introduce a general framework (HiPPO) for the online compression of continuous signals and discrete time series by projection onto polynomial bases. Given a measure that specifies the importance of each time step in the past, HiPPO produces an optimal solution to a natural online function approximation problem. As special cases, our framework yields a short derivation of the recent Legendre Memory Unit (LMU) from first principles, and generalizes the ubiquitous gating mechanism of recurrent neural networks such as GRUs. This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale. HiPPO-LegS enjoys the theoretical benefits of timescale robustness, fast updates, and bounded gradients. By incorporating the memory dynamics into recurrent neural networks, HiPPO RNNs can empirically capture complex temporal dependencies. On the benchmark permuted MNIST dataset, HiPPO-LegS sets a new state-of-the-art accuracy of 98.3%. Finally, on a novel trajectory classification task testing robustness to out-of-distribution timescales and missing data, HiPPO-LegS outperforms RNN and neural ODE baselines by 25-40% accuracy.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “Similarly, we also did not consider state-space models (Gu et al., 2020).”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “[33] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.”
It's Raw! Audio Generation with State-Space Models (2022, background): “All three HiPPO matrices from [12] are unitarily equivalent to a matrix of the form Furthermore, all entries of Λ have real part 0 (for HiPPO-LegT and HiPPO-LagT) or − (for HiPPO-LegS).”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, methodology): “Gu et al. [16] also consider a case of HiPPO corresponding to the generalized Laguerre polynomials that generalizes the above HiPPO-LagT case.”
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation (2024, methodology): “Specifically, the state matrix is crafted and initialized with High-Order Polynomial Projection Operator (HIPPO) [13], allowing to building deep sequence models with rich capability and efficient long-range reasoning ability.”
Is Mamba Effective for Time Series Forecasting? (2024, neutral): “Some researchers have also considered the application of Transformer-based time series forecasting models in specific domains, such as piezometric level prediction [42], forecasting crude oil returns [1], predicting the power generation by solar panels [49], etc.”
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation (2021, neutral): “These classes of matrices, and their generalization, have been used in machine learning to replace dense matrices in fully connected, convolutional, and recurrent layers [32, 55, 61].”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, methodology): “…LSSLs using a carefully chosen class of structured matrices A , such that (i) these matrices generalize prior work on continuous-time memory [24] and mathematically capture long dependencies with respect to a learnable family of measures, and (ii) with new algorithms, LSSLs with these…”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “Inspired by interesting approaches involving continuous-time representation for recurrent neural networks (Voelker et al., 2019), Gu et al. (2020) recently provided an alternative view on the vanishing gradient problem: one can design linear continuous-time state-space models (SSMs), of the form…”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “Many generalizations of these transforms have been used in machine learning to replace dense weight matrices (Sindhwani et al., 2015; Thomas et al., 2018; Gu et al., 2020).”
Simplified State Space Layers for Sequence Modeling (2022, methodology): “Each SSM leverages the HiPPO framework [15] by initializing with specially constructed state matrices.”
VMamba: Visual State Space Model (2024, methodology): “Inspired by continuous state space models in control systems, combined with HiPPO [15] initialization, LSSL [18] showcases the potential in handling long range dependency problems.”
Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (2022, methodology): “The ability to capture long-term dependencies relates to a particular initialization of A ∈ RN×N according to HiPPO theory [18, 20].”
RS-Mamba for Large Remote Sensing Image Dense Prediction (2024, methodology): “The integration of SSM into deep learning was catalyzed by the introduction of the highest polynomial powered operator (HiPPO) initialization method [31], which significantly improved the models’ ability to capture long-range dependencies.”
Diagonal State Spaces are as Effective as Structured State Spaces (2022, background): “Despite S4’s achievements, its design is complex and is centered around the HiPPO theory, which is a mathematical framework for long-range modeling [VKE19, GDE ` 20, GJG ` 21].”
PointMamba: A Simple State Space Model for Point Cloud Analysis (2024, background): “Linear state space equations [15, 18], combined with deep learning, offer a compelling approach for modeling sequential data, presenting an alternative to CNNs or Transformers.”
Multiwavelet-based Operator Learning for Differential Equations (2021, background): “[34] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.”
Hungry Hungry Hippos: Towards Language Modeling with State Space Models (2022, positive): “The diagonal SSM constrains A to be diagonal and initializes it from the diagonal version of HiPPO (S4D [26]).”
MambaIR: A Simple Baseline for Image Restoration with State-Space Model (2024, background): “In detail, the discretized state space equations in Mamba can be formalized into a recursive form and can model very long-range dependencies when equipped with specially designed structured reparameterization [23].”