Learning Long Term Dependencies via Fourier Recurrent Units

By Jiong Zhang, Yibo Lin, Zhao Song, Inderjit S. Dhillon. 2018

45 citations recorded by the source corpus.

It is a known fact that training recurrent neural networks for tasks that have long term dependencies is challenging. One of the main reasons is the vanishing or exploding gradient problem, which prevents gradient information from propagating to early layers. In this paper we propose a simple recurrent architecture, the Fourier Recurrent Unit (FRU), that stabilizes the gradients that arise in its training while giving us stronger expressive power. Specifically, FRU summarizes the hidden states $h^{(t)}$ along the temporal dimension with Fourier basis functions. This allows gradients to easily reach any layer due to FRU's residual learning structure and the global support of trigonometric functions. We show that FRU has gradient lower and upper bounds independent of temporal dimension. We also show the strong expressivity of sparse Fourier basis, from which FRU obtains its strong expressive power. Our experimental study also demonstrates that with fewer parameters the proposed architecture outperforms other recurrent architectures on many tasks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Christopher R'e
2. Tri Dao
3. Atri Rudra
4. Albert Gu
5. Stefano Ermon
6. Sergey Levine
7. Pieter Abbeel
8. Yang Song

In Statistical and Nonlinear Physics

1. Christopher R'e
2. Stefano Ermon
3. Albert Gu
4. Sergey Levine
5. Jiaming Song
6. Arash Vahdat
7. Pieter Abbeel
8. Yang Song

In Model Architectures

1. Tri Dao
2. Christopher R'e
3. Atri Rudra
4. Albert Gu
5. Stefano Ermon
6. Daniel Y. Fu
7. Karan Goel
8. Ali Hatamizadeh

Citations in the corpus

• A Convergence Theory for Deep Learning via Over-Parameterization (2018, neutral): “We choose τ = 1 Ω(L logm) which is similar as previous work [58].”
• On the Convergence Rate of Training Recurrent Neural Networks (2018, neutral): “ariations have been designed over past decades, including LSTM [29], Bidirectional RNN [60], Bidirectional LSTM [22], gate recurrent unit [11], statistical recurrent unit [46], Fourier recurrent unit [79]. For a more detailed survey, we refer the readers to Salehinejad et al. [57]. 1.1 Our Question ”
• HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, methodology): “5), contains proofs of their derivations, and derives additional HiPPO formulas for other bases such as Fourier (recovering the Fourier Recurrent Unit [79]) and Chebyshev.”