Training Continuous Space Language Models: Some Practical Issues

By Hang Le, Alexander Allauzen, Guillaume Wisniewski, François Yvon. 2010

39 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Li Deng
2. Yoshua Bengio
3. Holger Schwenk
4. Dong Yu
5. Kyunghyun Cho
6. Jianfeng Gao
7. Graham Neubig
8. Hermann Ney

Citations in the corpus

• Deep Learning: Methods and Applications (2014, background): “Subsequent work on NNLM with “deep” architectures can be found in [205, 207, 208, 245, 247, 248].”
• A tutorial survey of architectures, algorithms, and applications for deep learning (2014, background): “In another type of deep architecture that has created substantial impact in image recognition, Le et al. (2012) reported excellent results using a generative model based on sparse autoencoders in a largely un-supervised framework.”
• Continuous Space Translation Models for Phrase-Based Statistical Machine Translation (2012, methodology): “The continuous space language model (CSLM) was very successfully applied to large vocabulary speech recognition, and more recently to SMT, e.g. (Schwenk et al., 2006; Le et al., 2010; Zamora-Martínez et al., 2010; Schwenk et al., 2012).”
• Structured Output Layer Neural Network Language Models for Speech Recognition (2013, background): “Words in the shortlist are regarded as a special case since each represents its own class without sub-clustering (the tree depth is equal to 1 in this case).”
• Structured Output Layer neural network language model (2011, methodology): “Recent work [9] showed the impact of initialization and proposed new faster training schemes.”
• An Overview of Deep-Structured Learning for Information Processing (2011, background): “More recent work on neural network LM with deep architectures can be found in (Le et al., 2010, 2011; Mikolov et al., 2010).”
• Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation (2012, methodology): “…recently, the CSLM was also successfully applied to statistical machine translation (Schwenk et al., 2006; Schwenk and Est`eve, 2008; Schwenk, 2010; Le et al., 2010) During the last years, several extensions were proposed in the literature, for instance: • Mikolov and his colleagues are working on…”
• LSTM Neural Networks for Language Modeling (2012, methodology): “1As opposed to our LSTM version, in [14] the gating units do not receive the activations of the previous hidden layer • As training criterion the cross entropy error is used which is equivalent to maximum likelihood.”