A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property

By Jianshu Chen, Li Deng. 2013

27 citations recorded by the source corpus.

We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via auto-regression (AR) and moving-average (MA), respectively. The focus of this paper is a primal-dual training method that formulates the learning of the RNN as a formal optimization problem with an inequality constraint that provides a sufficient condition for the stability of the network dynamics. Experimental results demonstrate the effectiveness of this new method, which achieves 18.86% phone recognition error on the TIMIT benchmark for the core test set. The result approaches the best result of 17.7%, which was obtained by using RNN with long short-term memory (LSTM). The results also show that the proposed primal-dual training method produces lower recognition errors than the popular RNN methods developed earlier based on the carefully tuned threshold parameter that heuristically prevents the gradient from exploding.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Li Deng
3. Yoshua Bengio
4. Xiaodong He
5. Dong Yu
6. Mohit Bansal
7. Kyunghyun Cho
8. Richard Socher

In Signal Processing

1. Li Deng
2. Yoshua Bengio
3. Dong Yu
4. Kyunghyun Cho
5. Jianshu Chen
6. Xiaodong He
7. Jinyu Li
8. Jianfeng Gao

Citations in the corpus

• Deep Learning: Methods and Applications (2014, methodology): “More recently, a recurrent neural network (RNN) is used as the “backend” recognizer receiving the high-dimensional, DNN-derived features as the input without dimensionality reduction [48, 85].”
• Modeling Interestingness with Deep Neural Networks (2014, positive): “One potentially effective model for such a purpose is based on the architecture of recurrent neural networks (e.g., Mikolov et al. 2010; Chen and Deng, 2014), which can be incorporated into the deep semantic model proposed in this paper.”
• Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval (2015, methodology): “The main idea of using RNN for sentence embedding is to find a dense and low dimensional semantic representation by sequentially and recurrently processing each word in a sentence and mapping it into a low dimensional vector.”
• How to Construct Deep Recurrent Neural Networks (2013, background): “Recently, Chen and Deng (2013) reported that a better speech recognition performance could be achieved by employing this strategy, although they did not jointly train the deep input-to-hidden function together with other parameters of an RNN.”
• Sequence classification using the high-level features extracted from deep neural networks (2014, methodology): “optimization problem with an inequality constraint imposed to guarantee the stability of the RNN dynamics (see [3]).”
• A Primal Dual Formulation For Deep Learning With Constraints (2019, background): “Chen and Deng [2013] have employed a primal-dual based formulation for optimizing with constraints in deep models, but their constraints are specified over the weights in a recurrent neural network and are only concerned with imparting stability to the overall learning algorithm.”