Rank, Trace-Norm and Max-Norm

By Nati Srebro, Adi Shraibman. 2005

409 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Electrical and Electronic Engineering, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Maxout Networks (2013, methodology): “We regularized the model with dropout and by imposing a constraint on the norm of each weight vector, as in (Srebro & Shraibman, 2005).”
Pylearn2: a machine learning research library (2013, background): “Many Pylearn2 models impose nonnegativity constraints on parameters that represent, for e xample, the conditional variance of some random variable, and most Pylearn2 neural network layers su pport max norm constraints on the weights [48].”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “The idea of normalizing the weight vector has been proposed before (e.g. [27]) but earlier work typically still performed optimization in the w -parameterization, only applying the normalization after each step of stochastic gradient descent.”
Initialization and Regularization of Factorized Neural Layers (2021, background): “…problem: for example, if we consider the case of d “ 0 then this regularizer is in fact an upper bound on the nuclear norm of the recomposed matrix UV T (Srebro & Shraibman, 2005, Lemma 1): In fact, Figure 1 shows that for weight-decay this upper bound is tight throughout the training of…”
An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks (2013, methodology): “The hyperparameters we search over include the magnitude of the maxnorm constraint (Srebro & Shraibman, 2005) for each layer, the method used to initialize the weights for each layer and any hyper-parameters associated with such method, the initial biases for each layer, the parameters controlling…”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Normalizing weight tensors has been generally explored for speeding up the convergence of the optimization process [7,55,60].”
Exploring Generalization in Deep Learning (2017, methodology): “Also norms such as the trace norm and max norm are considered as sensible inductive biases in matrix factorization and are often more appropriate than parameter-counting measures such as the rank [27, 28].”
Few-Shot Learning via Learning the Representation, Provably (2020, neutral): “See reference e.g. Srebro and Shraibman (2005).”
Joint covariate selection and joint subspace selection for multiple classification problems (2010, methodology): “In particular, the trace norm, used by Argyriou et al. (2008), has been the focus of a recent theoretical literature Srebro et al. (2005a); Srebro and Shraibman (2005); Bach (2008); Recht et al. (2008).”
Path-SGD: Path-Normalized Optimization in Deep Neural Networks (2015, methodology): “φp(w) = min w̃∼w ( μp,∞(w̃) )d (1)This definition of max-norm is a bit different than the one used in the context of matrix factorization [13].”
Question Answering over Freebase with Multi-Column Convolutional Neural Networks (2015, methodology): “Moreover, the max-norm regularization (Srebro and Shraibman, 2005; Srivastava et al., 2014) is used for the column vectors of parameter matrices.”
Excess risk bounds for multitask learning with trace norm regularization (2012, methodology): “A linear map A : H → H is called an operator and self-adjoint if A ∗ = A and nonnegative (or positive) if it is self-adjoint and h Ax, x i ≥ 0 (or Ax, x > 0) for all x ∈ H , x = 0, in which case we write A 0 (or A ≻ 0). the operators h i 6 (cid:23) We use ” ” to denote order induced by the cone of…”
Implicit Regularization in Matrix Factorization (2017, background): “The emerging story is that gradient descent biases us to a low nuclear norm solution, and we already know how having low nuclear norm can ensure generalization [17, 6] and minimizing the nuclear norm ensures reconstruction [15, 5].”
Explicit Inductive Bias for Transfer Learning with Convolutional Networks (2018, background): “Max-norm regularization (Srebro & Shraibman 2005) is a hard constraint on the L 2 norm that was found especially helpful when using dropout (Srivastava et al. 2014).”
Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering (2010, background): “A simple option is to add a regularization term based on the l2 norm of these factors [19].”
Pretraining Convolutional Neural Networks for Visual Recognition (2016, background): “By setting a maximum kernel norm value, a small constant is reduced from the weight that achieves the maximum value [38].”
Privacy-preserving deep learning (2015, methodology): “Max-norm has been used for collaborative filtering [46] and deep learning [47].”
Training deep neural networks with low precision multiplications (2014, methodology): “We regularize the model with dropout and a constraint on the norm of each weight vector, as in (Srebro and Shraibman, 2005).”