Automatic node selection for Deep Neural Networks using Group Lasso regularization
By Tsubasa Ochiai, Shigeki Matsuda, Hideyuki Watanabe, Shigeru Katagiri. 2016
25 citations recorded by the source corpus.
We examine the effect of the Group Lasso (gLasso) regularizer in selecting the salient nodes of Deep Neural Network (DNN) hidden layers by applying a DNN-HMM hybrid speech recognizer to TED Talks speech data. We test two types of gLasso regularization, one for outgoing weight vectors and another for incoming weight vectors, as well as two sizes of DNNs: 2048 hidden layer nodes and 4096 nodes. Furthermore, we compare gLasso and L2 regularizers. Our experiment results demonstrate that our DNN training, in which the gLasso regularizer was embedded, successfully selected the hidden layer nodes that are necessary and sufficient for achieving high classification power.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jiayu Yao
- Finale Doshi-Velez
- Soumya Ghosh
- Michael C. Hughes
- Maurizio Zazzi
- Mike Wu
- Volker Roth
- Sonali Parbhoo
Citations in the corpus
- Beyond Sparsity: Tree Regularization of Deep Models for Interpretability (2017, methodology): “Edge and node regularization is commonly used to improve prediction accuracy (Drucker and Le Cun 1992; Ochiai et al. 2017), and recently Hu et al. (2016) improve prediction accuracy by training neural networks so that predictions match a small list of known domain-specific first-order logic rules.”
- Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, background): “Closer to our work in spirit, (Ochiai et al., 2016; Scardapane et al., 2017; Alvarez & Salzmann, 2016) and (Murray & Chiang, 2015) who use group sparsity to prune groups of weights—e.”