Long Expressive Memory for Sequence Modeling

By T. Konstantin Rusch, Siddhartha Mishra, N. Benjamin Erichson, Michael W. Mahoney. 2021

60 citations recorded by the source corpus.

We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks with very long-term dependencies, and it is sufficiently expressive to be able to learn complicated input-output maps. To derive LEM, we consider a system of multiscale ordinary differential equations, as well as a suitable time-discretization of this system. For LEM, we derive rigorous bounds to show the mitigation of the exploding and vanishing gradients problem, a well-known challenge for gradient-based recurrent sequential learning methods. We also prove that LEM can approximate a large class of dynamical systems to high accuracy. Our empirical results, ranging from image and time-series classification through dynamical systems prediction to speech recognition and language modeling, demonstrate that LEM outperforms state-of-the-art recurrent neural networks, gated recurrent units, and long short-term memory models.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Lei He
3. Sheng Zhao
4. Yanqing Liu
5. Tao Qin
6. Tie-Yan Liu
7. Rui Xue
8. Frank K. Soong

In Statistical and Nonlinear Physics

1. Yanqing Liu
2. Sheng Zhao
3. Xu Tan
4. Lei He

Citations in the corpus

• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders (2022, methodology): “The decoder adopts the same architecture as the generator in HiFi-GAN [15], with a bidirectional Long Expressive Memory (LEM) [16] layer at upsampling stage for a stable learning of long-term sequential dependencies.”