Investigations on hessian-free optimization for cross-entropy training of deep neural networks

33 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Deep Learning: Methods and Applications (2014, methodology): “[390] investigated the Hessian-free optimization method for training the DNN with the cross-entropy objective and empirically analyzed the properties of the method.”
Hierarchical Bayesian combination of plug-in maximum a posteriori decoders in deep neural networks-based speech recognition and speaker adaptation (2017, methodology): “The DNN was trained with an initial learning rate of 0.008 using the cross-entropy (Boer et al., 2005; Wiesler et al., 2013) objective function.”
Mean-normalized stochastic gradient for large-scale deep learning (2014, methodology): “, algorithms where the gradient is computed on the full dataset, for example LBFGS [5, 6], Rprop [7], or the Hessian-Free algorithm [8, 9].”
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs (2014, methodology): “…model structure and training approach, e.g. [10, 11] where the network was factored into a hierarchy; low-rank approximations [12, 13]; second-order methods (“Hessian-Free”) [14, 15]; model averaging [16]; or ADMM which clev-erly tweaks the objective function for better parallelizability [17, 18].”