Efficient Subsampling for Training Complex Language Models

By Puyang Xu, Asela Gunawardana, Sanjeev Khudanpur. 2011

28 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

One billion word benchmark for measuring progress in statistical language modeling (2013, methodology): “The Binary MaxEnt model was proposed in (Xu et al., 2011) and aims to avoid the expensive probability normalization during training by using independent binary predictors.”
Variational Autoencoders for Collaborative Filtering (2018, neutral): “d multinomial likelihood is closer — this is understandable since multinomial likelihood can be approximated by individual binary logistic likelihood, a strategy commonly adopted in language modeling [32, 52]. We wish to emphasize that the choice of likelihood remains datadependent. For the task of ”
Strategies for training large scale neural network language models (2011, methodology): “Although the subsampling trick can be used to claim that the neural network model has seen all training data at least once, simple subsampling techniques lead to severe performance degradation, against a model that is trained on all data - a more advanced subsampling technique has been recently intr”
Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation (2012, background): “• alternative sampling in large corpora (Xu et al., 2011)”
Statistical Language Models Based on Neural Networks (2012, background): “Although the sampling trick can be used to claim that the neural network model has seen all the training data at least once, simple sampling techniques lead to severe performance degradation, against a model that is trained on all data - a more advanced sampling technique has been recently introduce”