Nonparametric Neural Networks

By George Philipp, Jaime G. Carbonell. 2017

21 citations recorded by the source corpus.

Automatically determining the optimal size of a neural network for a given task without prior information currently requires an expensive global search and training many networks from scratch. In this paper, we address the problem of automatically finding a good network size during a single training cycle. We introduce *nonparametric neural networks*, a non-probabilistic framework for conducting optimization over all possible network sizes and prove its soundness when network growth is limited via an L_p penalty. We train networks under this framework by continuously adding new units while eliminating redundant units via an L_2 penalty. We employ a novel optimization algorithm, which we term *adaptive radial-angular gradient descent* or *AdaRad*, and obtain promising results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wujie Wen
2. Hai Helen Li
3. Yiran Chen
4. Yuxiong He
5. Samyam Rajbhandari
6. Minjia Zhang
7. Wenhan Wang
8. Peter Richtárik

In Computer Vision and Pattern Recognition

1. Wujie Wen
2. Hai Helen Li
3. Yiran Chen
4. Samyam Rajbhandari
5. Yuxiong He
6. Minjia Zhang
7. Greg Mori
8. Georgios Georgiadis

In Model Architectures

1. Wujie Wen
2. Yuxiong He
3. Samyam Rajbhandari
4. Hai Helen Li
5. Wenhan Wang
6. Yiran Chen
7. Minjia Zhang

Citations in the corpus

• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “Structure pruning [37, 18, 17, 27, 12, 24, 23, 7, 13, 10] and growing [9, 38, 27, 7] were proposed for different goals, such as efficient inference [37, 18, 17, 12, 24, 23, 7, 13, 10], lifelong learning [38] and model adaptation [9, 27].”
• Lifelong Learning with Dynamically Expandable Networks (2017, neutral): “Recently, Philipp & Carbonell (2017) propose a nonparametric neural network model which not only learns to minimize the loss but also find the minimum dimensionality of each layer that can reduce the loss, with the assumption that each layer has infinite number of neurons.”
• Learning Intrinsic Sparse Structures within Long Short-term Memory (2017, methodology): “Philipp & Carbonell (2017) uses group Lasso regularization to dynamically add and eliminate neurons in FNNs. Cortes et al. (2017) incrementally adds sub-networks to current networks by incrementally reducing the objective function.”