Learning Compact Neural Networks with Regularization

By Samet Oymak. 2018

39 citations recorded by the source corpus.

We study the impact of regularization for learning neural networks. Our goal is speeding up training, improving generalization performance, and training compact models that are cost efficient. Our results apply to weight-sharing (e.g.~convolutional), sparsity (i.e.~pruning), and low-rank constraints among others. We first introduce covering dimension of the constraint set and provide a Rademacher complexity bound providing insights on generalization properties. Then, we propose and analyze regularized gradient descent algorithms for learning shallow networks. We show that problem becomes well conditioned and local linear convergence occurs once the amount of data exceeds covering dimension (e.g.~\# of nonzero weights). Finally, we provide insights on layerwise training of deep models by studying a random activation model. Our results show how regularization can be beneficial to overcome overparametrization.

Subfields: Artificial Intelligence, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

A Convergence Theory for Deep Learning via Over-Parameterization (2018, methodology): “There is sequence of work about one-hidden-layer (multiple neurons) CNN (Brutzkus & Globerson, 2017; Du et al., 2018b; Goel et al., 2018; Oymak, 2018; Zhong et al., 2017a).”
Deep Declarative Networks (2019, background): “While the solution, and hence gradients, can be expressed in closed-form, it may be desirable to use other Lp-spheres or balls, for regularization, sparsification, or to improve generalization performance [43], as well as for adversarial robustness.”
Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path? (2018, background): “An equally important question to understanding the convergence behavior of optimization algorithms for overparameterized models is understanding their generalization capabilities this is the subject of a few interesting recent papers (Arora et al., 2018b; Bartlett et al., 2017; Belkin et al., 2018a;”
Communication-Efficient Edge AI: Algorithms and Systems (2020, background): “…[61], randomized sketching [62], [63], [64], network pruning [65], [66], [67], [27], [68], [69], [70], [71], sparse regularization [72], [73], [74], and structural matrix designing for minimizing communication bandwidth [75], [76], [77], [78], [79], [80], [81], [82], [83]…”