PAC-Bayesian learning of linear classifiers

By Pascal Germain, Alexandre Lacasse, François Laviolette, Mario Marchand. 2009

268 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

PAC-Bayesian Theory Meets Bayesian Inference (2016, neutral): “Theorem 1, due to Catoni [9], has been used to derive or study learning algorithms [12, 22, 34, 37].”
A PAC-Bayesian bound for Lifelong Learning (2013, methodology): “In this case the expected empirical error of the Gibbs classifier is given by the following expression (Germain et al., 2009; Langford & Shawe-Taylor, 2002)”
A PAC-Bayesian Tutorial with A Dropout Bound (2013, background): “PAC-Bayesian generalization bounds [11, 12, 10, 7, 4, 6] govern the performance (loss) when stochastically selecting rules from a “posterior” distribution.”
Tighter PAC-Bayes bounds through distribution-dependent priors (2013, unknown): “The following is a generalization of (Germain et al., 2009, Theorem 2.1) and is proved using the same sequence of arguments.”
A New PAC-Bayesian Perspective on Domain Adaptation (2015, unknown): “…one could try to reduce its volume while taking care to preserve a good compromise between d T ( ⇢ ) and e S ( ⇢ ) , using a representation learning approach, i.e. , by projecting source and target examples into a new common space, as done for example by Chen et al. (2012); Ganin et al. (2016).”
PAC-bayes bounds with data dependent priors (2012, background): “The tightest bounds for practical applications appear to be the PAC-Bayes bound (McAllester, 1999; Langford and Shawe-Taylor, 2002; Catoni, 2007) and in particular the form given in Seeger (2002), Langford (2005) and Germain et al. (2009).”
PAC-Bayesian Bounds based on the Rényi Divergence (2016, unknown): “So far, most algorithms derived from PAC-Bayesian bounds are KL-regularized [ e.g. Germain et al., 2009, Parrado-Hernández et al., 2012, Pentina and Lampert, 2015, Alquier et al., 2015].”
PAC-Bayesian Analysis of Co-clustering and Beyond (2010, methodology): “Co-clustering is a widely used method for analysis of data in matrix form by simultaneous clustering of rows and columns of the matrix (Banerjee et al., 2007).”
Risk bounds for the majority vote: from a PAC-Bayesian analysis to a learning algorithm (2015, methodology): “The following theory is similar to the one used by Germain et al. (2011), in which two learning algorithms inspired by the PAC-Bayesian theory are compared : one regularized with the KL divergence, using a hyperparameter to control its weight, and one regularized by restricting the posterior distrib”
Robust Forward Algorithms via PAC-Bayes and Laplace Distributions (2014, background): “Germain et al [15] derived a specialized loss [13] plugged in a PAC-Bayes bound, and used quasi-uniform distributions over a finite set of classifiers.”
Stability and Generalization in Structured Prediction (2016, methodology): “PAC-Bayesian analysis is often accredited to McAllester (1998, 1999), and has been refined by a number of authors (e.g., Herbrich and Graepel, 2001; Langford and Shawe-Taylor, 2002; Seeger, 2002; Ambroladze et al., 2006; Catoni, 2007; Germain et al., 2009; Lever et al., 2010; Seldin et al., 2012).”
Chromatic PAC-Bayes Bounds for Non-IID Data (2009, background): “…Pac -Bayes bounds for linear classiﬁers: Ambroladze et al. (2007) showed that Pac -Bayes bounds are a viable route to do actual model selection; Germain et al. (2009) recently proposed to learn linear clas-siﬁers by directly minimizing the linear Pac -Bayes bound with conclusive results,…”
PAC-Bayesian Theory for Transductive Learning (2014, methodology): “As discussed in Germain et al. [2009], Theorem 1 is a generic tool to derive various inductive PAC-Bayesian bounds, as D can be any convex function.”
PAC-Bayes-Bernstein Inequality for Martingales and its Application to Multiarmed Bandits (2011, methodology): “The PAC-Bayesian analysis was applied to derive generalization bounds and new algorithms for linear classifiers and maximum margin methods (Langford and Shawe-Taylor, 2002; McAllester, 2003; Germain et al., 2009), structured prediction (McAllester, 2007), and clustering-based classification models (”
Learning Efficient Random Maximum A-Posteriori Predictors with Non-Decomposable Loss Functions (2013, methodology): “Interestingly, when incorporating perturbmax models to Bayesian loss minimization one would ultimately like to use the PAC-Bayesian risk [11, 19, 3, 20, 5, 10].”
PAC-Bayesian Collective Stability (2014, methodology): “PAC-Bayesian analysis was introduced by McAllester (1999) and later refined by a number of authors (e.g., Langford and Shawe-Taylor, 2002; Seeger, 2002; Ambroladze et al., 2006; Germain et al., 2009).”
Robust Optimization using Machine Learning for Uncertainty Sets (2014, background): “As shown by Germain et al. [2009], for a certain choice of the metric D the above theorem gives a bound on R(GQ) that is proportional to CnRS(GQ) + KL(Q||P ) where C is a prespecified constant.”