Convex Neural Networks

By Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, Patrice Marcotte. 2005

226 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Supermasks in Superposition (2020, background): “[1] Yoshua Bengio, Nicolas L Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte.”
Representing smooth functions as compositions of near-identity functions with implications for deep network optimization (2018, background): “ze residual networks with a countably inﬁnite number of parameters on each layer, for instance by sequentially adding units. Such algorithms have been studied for two-layer networks; see (Bach, 2014; Bengio et al., 2006; Lee et al., 1996). Indeed, standard denseness results (see the remarks after Th”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “…we know that it is suﬃcient to regularize only the output weights in order to constrain capacity: in the limit case of the number of hidden units going to inﬁnity, L2 regularization corresponds to Support Vector Machines (SVM) while L1 regularization corresponds to boosting (Bengio et al. , 2006a).”
A mean field view of the landscape of two-layer neural networks (2018, background): “[18] The convergence of the SGD process to the PDE model is an example of a phenomenon that is known in probability theory as propagation of chaos (23).”
In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning (2014, methodology): “Now we establish a connection between the ℓ 1 -regularized network (5), in which we learn the input to hidden connections, to convex NN (Bengio et al., 2005).”
Recovery Guarantees for One-hidden-layer Neural Networks (2017, background): “The works by (Bach, 2014) and (Bengio et al., 2005) are also in this direction.”
ℓ1-regularized Neural Networks are Improperly Learnable in Polynomial Time (2015, background): “While it is possible to transform the neural network training problem to a convex optimization problem involving an inﬁnite number of variables [5], the inﬁnitude of variables means that there is no longer a guarantee that the learning algorithm will terminate in polynomial time.”
Norm-Based Capacity Control in Neural Networks (2015, background): “Convex Neural Nets (Bengio et al., 2005) over inputs in R D are two-layer networks with a ﬁxed inﬁnite hidden layer consisting of all units with weights w ∈ G for some base class G ∈ R D , and a second ℓ 1 -regularized layer. the Since over ﬁnite data weights in the second layer can always be taken ”
Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks (2018, background): “The closeness of learned weights to initialization in the over-parametrized setting can be understood by considering the limiting case as the number of hidden units go to inﬁnity, as considered in Bengio et al. [5] and Bach [2].”
Benign overfitting in linear regression (2019, background): “[16] Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte.”
Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations (2019, background): “It was noted that when we consider NNs with an infinite number of neurons, then the network becomes more and more convex, and thus can be analyzed theoretically in the limit (Lee et al., 1996; Bengio et al., 2006; Mei et al., 2018; Chizat & Bach, 2018; Du et al., 2019a; Allen-Zhu et al., 2019).”
Adversarial Training Versus Weight Decay (2018, background): “Recently, there has been a renewed interest in developing knowledge in norm-based capacity control of linear and depth-limited networks (Bengio et al., 2006; Neyshabur et al., 2015) as an explanation for generalization in deep learning (Neyshabur et al., 2017; Tsuzuku et al., 2018).”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “Indeed, (Arora et al., 2018; Bengio et al., 2006; Bach, 2014; Janzamin et al., 2015) show global optimal approximations, while other works have shown that networks based on sequential 1-hidden layer training can have a variety of guarantees under certain assumptions (Huang et al., 2017; Malach &…”
Neural Networks and Deep Learning (2018, unknown): “The relationship between boosting and certain types of neural architectures is pointed out in [32].”
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss (2020, unknown): “The F 1 -max-margin of the training set is denoted γ 1 and given by For ReLU networks, the variation norm deﬁned above does not a priori coincide with the variation norm as deﬁned by Bengio et al. (2006) and Bach (2017a).”
Learning Deep Architectures for AI (2007, methodology): “Hence, one hypothesis is that unsupervised pre-training he lps generalization by allowing for a ’better’ tuning of lower layers of a deep architecture. Although trai ning error can be reduced either by exploiting only the top layers ability to fit the training examples, bett er generalization is ach”