Riemannian metrics for neural networks
By Yann Ollivier. 2013
22 citations recorded by the source corpus.
Subfields: Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Yoshua Bengio
- James Martens
- Pascal Vincent
- Razvan Pascanu
- Roger Baker Grosse
- Karen Simonyan
- Stefano Soatto
- Thomas Unterthiner
Citations in the corpus
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The resulting formulas are similar to Definition 6 of Olivier (2013). Theorem 1.”
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature (2015, methodology): “udied are those which work by directly inverting a diagonal, block-diagonal, or low-rank approximation to the curvature matrix (e.g. Becker and LeCun, 1989; Schaul et al., 2013; Le Roux et al., 2008; Ollivier, 2013). In fact, a diagonal approximation of the Fisher information matrix is used within H”
- Natural Neural Networks (2015, background): “See [15, 14] for a recent overview of the topic.”
- From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification (2016, neutral): “…model (Bradley & Terry, 1952; Zadrozny, 2001; Menke & Martinez, 2008), the multinomial probit (Albert & Chib, 1993), the spherical softmax (Ollivier, 2013; Vincent, 2015; de Br´ebisson & Vincent, 2015), or softmax approximations (Bouchard, 2007), while theoretically or computationally…”
- Diagonal Rescaling For Neural Networks (2017, methodology): “Invoking the usual second order optimization arguments [Becker and LeCun, 1989, Ollivier, 2013, Desjardins et al., 2015, Marceau-Caron and Ollivier, 2016] leads to tuning the reparametrization with a simple diagonal or block-diagonal approximation of the inverse curvature matrix.”
- Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets (2014, methodology): “Acknowledgements: We wish to thank Yves Grandvalet for stimulating discussions, Çağlar Gülçehre for pointing us to [14], the developers of Theano [17, 18] and Blocks [19] for making these libraries available to build on, and NSERC and Ubisoft for their financial support.”