Acceleration of stochastic approximation by averaging

By Boris Polyak, Anatoli B. Juditsky. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Xception: Deep Learning with Depthwise Separable Convolutions (2016, methodology): “Additionally, all models were evaluated using Polyak averaging [13] at inference time.”
Going deeper with convolutions (2014, methodology): “Polyak averaging [13] was used to create the final model used at inference time.”
Pylearn2: a machine learning research library (2013, background): “This class supports extensions including Polyak averaging [41], momentum [44], and early stopping.”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “For ImageNet results in Table 2, we ﬁne-tuned at higher resolution: 512 for ViT-L/16 and 518 for ViT-H/14, and also used Polyak & Juditsky (1992) averaging with a factor of 0 .”
Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (2019, background): “One way to reduce the steady state risk, apart from using Polyak averaging [Polyak and Juditsky, 1992] or decreasing the learning rate (which will harm the rate of convergence), is to increase the batch size.”
WebGPT: Browser-assisted question-answering with human feedback (2021, methodology): “For BC and RM, we used Polyak–Ruppert averaging [Polyak and Juditsky, 1992], taking an exponentially-weighted moving average (EMA) of the weights of the model as the final checkpoint.”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “002 for optimization, in combination with Polyak averaging [22] in the form of an exponential moving average that averages parameters over approximately 10 epochs.”
Emerging Properties in Self-Supervised Vision Transformers (2021, positive): “Indeed, we observe that this teacher performs a form of model ensembling similar to Polyak-Ruppert averaging with an exponential decay [41, 48].”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, methodology): “The learning rate was held constant at 2 × 10 − 4 , and Polyak averaging [21] was applied over the parameters.”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “We demonstrate the AIQN-VAE using the CelebA dataset (Liu et al., 2015), at resolution 64 × 64 . to We modiﬁed an open source VAE implementation 4 simultaneously train the AIQN on the output of the VAE encoder, with Polyak averaging (Polyak & Juditsky, 1992) of the AIQN weights.”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, methodology): “On both datasets, we use Polyak averaging [10] (following [11]) over the training parameters.”
Improved Variational Inference with Inverse Autoregressive Flow (2016, methodology): “002 and Polyak averaging (Polyak and Juditsky, 1992) to compute the ﬁnal parameters.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “Evaluation was performed on an exponential moving average estimate of the parameters (Polyak & Juditsky, 1992) using a smoothing rate of 0.9999 to help reduce the effect of the learning rate not fully decaying to zero for the models trained on the subsampled datasets due to early stopping.”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “Polyak averaging [53] was applied to all checkpoints used for model”
The challenge of realistic music generation: modelling raw audio at scale (2018, positive): “We use Polyak averaging for evaluation [35].”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “We include most of the augmentation and regularization strategies of [57] in training, except for repeated augmentation [28] and EMA [41], which do not enhance performance.”