Information Geometry of the Gaussian Distribution in View of Stochastic Optimization
By Luigi Malagò, Giovanni Pistone. 2015
66 citations recorded by the source corpus.
We study the optimization of a continuous function by its stochastic relaxation, i.e., the optimization of the expected value of the function itself with respect to a density in a statistical model. We focus on gradient descent techniques applied to models from the exponential family and in particular on the multivariate Gaussian distribution. From the theory of the exponential family, we reparametrize the Gaussian distribution using natural and expectation parameters, and we derive formulas for natural gradients in both parameterizations. We discuss some advantages of the natural parameterization for the identification of sub-models in the Gaussian distribution based on conditional independence assumptions among variables. Gaussian distributions are widely used in stochastic optimization and in particular in model-based Evolutionary Computation, as in Estimation of Distribution Algorithms and Evolutionary Strategies. By studying natural gradient flows over Gaussian distributions our analysis and results directly apply to the study of CMA-ES and NES algorithms.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alessandro Achille
- Stefano Soatto
- Giovanni Paolini
- A. Dechant
- Sosuke Ito
- Luigi Montrucchio
- Giovanni Pistone
- Luigi Malagò
Citations in the corpus
- Where is the Information in a Deep Neural Network? (2019, methodology): “Following Martens (2014), we use a more robust second-order approximation by ignoring the second part of eq.”
- Stochastic Time Evolution, Information Geometry, and the Cramér-Rao Bound (2018, methodology): “In this case, we can compute the rate of Shannon entropy change σsys(t) = dtΣ(t) and the Fisher information explicitly [38],”
- Wasserstein Riemannian geometry of Gaussian densities (2018, background): “We do not discuss this issue here and refer to [1,4,24].”