Scalable stacking and learning for building deep architectures

By Li Deng, Dong Yu, John C. Platt. 2012

217 citations recorded by the source corpus.

Deep Neural Networks (DNNs) have shown remarkable success in pattern recognition tasks. However, parallelizing DNN training across computers has been difficult. We present the Deep Stacking Network (DSN), which overcomes the problem of parallelizing learning algorithms for deep architectures. The DSN provides a method of stacking simple processing modules in buiding deep architectures, with a convex learning problem in each module. Additional fine tuning further improves the DSN, while introducing minor non-convexity. Full learning in the DSN is batch-mode, making it amenable to parallel training over many machines and thus be scalable over the potentially huge size of the training data. Experimental results on both the MNIST (image) and TIMIT (speech) classification tasks demonstrate that the DSN learning algorithm developed in this work is not only parallelizable in implementation but it also attains higher classification accuracy than the DNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neural Architecture Search & AutoML

Citations in the corpus

Predicting Parameters in Deep Learning (2013, neutral): “Other approaches to distributed learning of neural networks involve training in batch mode [8], but these methods have not been scaled nearly as far as their online counterparts.”
Deep Learning: Methods and Applications (2014, background): “This chapter is based in part on the recent publications of [106, 110, 180, 181] with expanded discussions.”
Learning Continuous Phrase Representations for Translation Modeling (2014, methodology): “The objective function, which takes the form of the expected BLEU computed from N-best lists, is very different from the usual objective functions used in most existing architectures of neural networks, e.g., cross entropy (Hin-ton et al. 2012) or mean square error (Deng et al. 2012).”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “er increase the number of connections between layers. As opposed to the module identity shortcut, denselyconnected blocks concatenateeach layer’s outputs to the inputs of the next layers, similar to [Deng et al. 2012]. According to the authors, this type of connection induces better gradient propaga”
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval (2015, positive): “This work has been motivated by the earlier successes of deep learning methods in speech [41]–[45] and in semantic modelling [3], [10], [46], [47], and it adds further evidence for the effectiveness of these methods.”
Distributed optimization of deeply nested systems (2012, methodology): “Parallel implementation of MAC/QP Our parallel implementation of MAC/QP is extremely simple at present, yet it achieves large speedups (about 6× faster if using 12 processors), which are nearly linear as a function of the number of processors for all experiments, as shown in fig.”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “we can derive batch-mode gradient descent learning algorithmwhere the gradient takes the following form [108]:”
Convolutional Neural Networks for Speech Recognition (2014, background): “O. Abdel-Hamid and H. Jiang are with the Department of Electrical Engineering and Computer Science, Lassonde School of Engineering, York University, Toronto, ON M3J 1P3, Canada (e-mail: ossama@cse.yorku.ca; hj@cse. yorku.ca).”
Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community (2017, background): “Deng et al.(70) put forth a deep stacking network (DSN) that consists of specialized NNs (called modules), each of which has a single hidden layer.”
New types of deep neural network learning for speech recognition and related applications: an overview (2013, background): “Three main factors were responsible for the recent emergence of neural networks as high-quality acoustic models: (1) making the networks deeper makes them more powerful, hence deep neural networks (DNN); 2) initializing the weights sensibly [24][43][16][52] and using much faster hardware makes it po”
Learning with Recursive Perceptual Representations (2012, background): “Another successful stacking architecture reported in [10, 11] uses supervised information for stacking where the basic module is a simplified form of multilayer perceptron where the output units are linear and the hidden units are sigmoidal nonlinear.”
Recent advances in deep learning for speech research at Microsoft (2013, background): “…researchers at University of Toronto and other organizations, we at Microsoft have developed deep learning technology that has successfully replaced Gaussian mixtures for speech recognition and feature coding at an increasingly larger scale (e.g., [24][19][53][39][7][8][44][54][13][56][30][48]).”
Deep Neural Networks for Acoustic Modeling in Speech Recognition (2012, methodology): “Some recent atte mpts are described in [52], [57].”
Towards deeper understanding: Deep convex networks for semantic utterance classification (2012, background): “DCN is shown to be superior to DBN, not only in terms of accuracy, but also in training scalability and efficiency [11, 12].”
Acoustic Modeling With Hierarchical Reservoirs (2013, background): “Due to this, alternative deep architectures like Deep Convex Networks [44] or Deep Stacking Networks [12], [13] were proposed recently to overcome the scalability challenge of DBNs by solving convex learning problems in each module, allowing a faster parallel batch training.”