A Convergence Analysis of Log-Linear Training

By Simon Wiesler, Hermann Ney. 2011

78 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, background): “It has been long known (LeCun et al., 1998b; Wiesler & Ney, 2011) that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated.”
Neural Networks and Deep Learning (2018, unknown): “Evaluations and analysis of the effect of feature preprocessing on neural network learning may be found in [278, 532].”
Investigations on an EM-Style Optimization Algorithm for Discriminative Training of HMMs (2013, background): “Optimal pre-processing and normalization of the features may help as well [43].”
Mean-normalized stochastic gradient for large-scale deep learning (2014, methodology): “In this work, we develop mean-normalized SGD (MN-SGD), a new stochastic second-order algorithm which uses analytic results [13, 14] about the structure of the objective function which is optimized in training.”
Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning (2017, background): “This step is known to make learning easier for neural networks (46, 47).”
Whitening and Coloring Batch Transform for GANs (2018, background): “In fact, feature normalization makes the loss landscape smoother (Santurkar et al. (2018); Kohler et al. (2018)), making the gradient-descent weight updates closer to Newton updates (Huang et al. (2018); LeCun et al. (2012); S. Wiesler (2011)).”
Learning Trans-Dimensional Random Fields with Applications to Language Modeling (2018, methodology): “by normalizing the means and variances of features [33].”
Automatically identifying wild animals in camera trap images with deep learning (2017, background): “This step is known to make learning easier for neural networks (43, 44).”