Recent Advances in Recurrent Neural Networks

By Hojjat Salehinejad, Julianne Baarbe, Sharan Sankar, Joseph Barfett, Errol Colak, Shahrokh Valaee. 2017

744 citations recorded by the source corpus.

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. The fundamentals and recent advances are explained and the research challenges are introduced.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuanzhi Li
2. Zeyuan Allen-Zhu
3. Quanquan Gu
4. Zhao Song
5. Simon Shaolei Du
6. Jason D. Lee
7. Jing Zhao
8. Shiliang Sun

In Signal Processing

1. Zeyuan Allen-Zhu
2. Zehui Cao
3. Zhao Song
4. Donald B. Percival
5. A. Sherstinsky
6. Shiliang Sun
7. Yuanzhi Li
8. Han Zhu

Citations in the corpus

• On the Convergence Rate of Training Recurrent Neural Networks (2018, methodology): “This is referred to as deep RNN [45].”
• Using labeled data to evaluate change detectors in a multivariate streaming environment (2009, positive): “Furthermore, since there are many small ice blocks in the river ice images, a learnable upsampling strategy DUpsampling is utilized to replace the commonly used bilinear interpolation to increase the segmentation accuracy, especially for small ice blocks.”
• Can SGD Learn Recurrent Neural Networks with Provable Generalization? (2019, background): “Recurrent neural networks (RNNs) is one of the most popular models in sequential data analysis [25].”
• A Survey of Optimization Methods From a Machine Learning Perspective (2019, methodology): “It was generally considered that the use of first-order methods to optimize RNN was not effective, because the SGD and its variant methods were difficult to learn long-term dependencies in sequence problems [87], [92], [163].”
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “In all sources to date, one or more of the elements in the above list is not addressed2 [5, 13, 14, 15, 16, 24, 26, 27, 32, 34, 35, 36, 40, 42, 50, 55, 56, 66, 68, 69, 78].”