Learning the Structure of Deep Architectures Using L1 Regularization
By Praveen Kulkarni, Joaquin Zepeda, Frédéric Jurie, Patrick Pérez, Louis Chevallier. 2015
22 citations recorded by the source corpus.
We present a method that formulates the selection of the structure of a deep architecture as a penalized, discriminative learning problem. Up to now, the structure of deep architectures has been fixed by hand, and only the weights are learned using discriminative learning. Our work is a first attempt towards a more formal method of deep structure selection. We consider architectures consisting only of fully-connected layers, and our approach relies on diagonal matrices inserted between subsequent layers. By including an L1 norm of the diagonal entries of said matrices as a regularization penalty, we force the diagonals to be sparse, accordingly selecting the effective number of rows (respectively, columns) of the corresponding layer's (next layer's) weights matrix. We carry out experiments on a standard dataset and show that our method succeeds in selecting the structure of deep architectures of multiple layers. One variant of our architecture results in a feature vector of size as little as $36$, while retaining very high image classification performance.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jakob Verbeek
- Quoc V. Le
- Christian Haase-Schuetz
- Werner Wiesbeck
- Di Feng
- Graham W. Taylor
- Klaus C. J. Dietmayer
- Saurabh Saxena
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Cordelia Schmid
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Alan L. Yuille
- Jitendra Malik
Citations in the corpus
- Convolutional Neural Fabrics (2016, methodology): “Finally, we analyze the number of response maps, parameters, and activations of fabrics in Section 3.3.”
- Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, methodology): “Another way to optimize the network structure is by regularization, such as l1 regularization [230] and stochastic regularization [231], [232].”
- Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, methodology): “Kulkar-ni et al. [86] implemented a method of learning the structure of DNNs via deterministic regularization.”