Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation

By Holger Schwenk, Anthony Rousseau, Mohammed Attik. 2012

105 citations recorded by the source corpus.

Subfields: Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “LMs have played a key role in traditional NLP tasks such as speech recognition (Mikolov et al., 2010; Arisoy et al., 2012), machine translation (Schwenk et al., 2012; Vaswani et al.)”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “, 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu & Blunsom, 2015).”
Representation Learning: A Review and New Perspectives (2012, methodology): “Similar models have been applied in statistical machine translation (Schwenk et al., 2012; Le et al., 2013), improving perplexity and BLEU scores.”
Efficient softmax approximation for GPUs (2016, background): “This problem is key to natural language processing, with applications in machine translation (Schwenk et al., 2012; Sutskever et al., 2014; Vaswani et al., 2013) or automatic speech recognition (Graves et al., 2013; Hinton et al., 2012).”
Joint Language and Translation Modeling with Recurrent Neural Networks (2013, background): “Recently, several feed-forward neural network-based language and translation models have achieved impressive accuracy improvements on statistical machine translation tasks (Allauzen et al., 2011; Le et al., 2012b; Schwenk et al., 2012).”
Learning Continuous Phrase Representations for Translation Modeling (2014, methodology): “…the CPTM is learning the continuous representation of a phrase that is effective for SMT. Motivated by recent studies on continuous-space language models (e.g., Bengio et al. 2003; Mikolov et al. 2011; Schwenk et al., 2012), we use a neural network to project a word vector to a feature vector.”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, methodology): “…include restricting the vocabulary of the NPLM to a shortlist and reverting to a traditional n -gram LM for other words (Schwenk, 2004), and limiting the number of training examples using resampling (Schwenk and Gauvain, 2005) or selecting a subset of the training data (Schwenk et al., 2012).”
Compositional Morphology for Word Representations and Language Modelling (2014, methodology): “Rescoring lattices with a CSLM proved to be beneﬁcial for ASR (Schwenk, 2004) and was subsequently applied to translation (Schwenk et al., 2006; Schwenk & Koehn, 2008), reaching training sizes of up to 500m words (Schwenk et al., 2012).”
Strategies for Training Large Vocabulary Neural Language Models (2015, background): “Neural network language models (Bengio et al., 2003; Mikolov et al., 2010) have gained popularity for tasks such as automatic speech recognition (Arisoy et al., 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2014).”
Topic Compositional Neural Language Model (2017, background): “It plays a key role in many traditional NLP tasks, ranging from speech recognition (Mikolov et al., 2010; Arisoy et al., 2012; Sriram et al., 2017), machine translation (Schwenk et al., 2012; Vaswani et al., 2013) to image captioning (Mao et al., 2014; Devlin et al., 2015).”
Factored recurrent neural network language model in TED lecture transcription (2012, methodology): “The training corpus is gradually increased by selecting sentences from the generaldomain corpus [17, 20].”
Knowledge Distillation for Recurrent Neural Network Language Modeling with Trust Regularization (2019, background): “Recurrent Neural Network based language models ( RNNLM s) have outperformed traditional N-gram based language models in many machine learning tasks, such as automatic speech recognition [1], machine translation [2] or text summarization [3].”
Accelerating Text Communication via Abbreviated Sentence Input (2021, background): “Other work has investigated data selection using cross-entropy or cross-entry difference between in-and out-of-domain datasets (Axelrod et al., 2011; Moore and Lewis, 2010; Schwenk et al., 2012; Rousseau, 2013; Mansour et al., 2011; Vertanen and Kristensson, 2011b).”
An Effective Neural Network Model for Graph-based Dependency Parsing (2015, unknown): “2003; Collobert et al., 2011; Schwenk et al., 2012; Mikolov et al., 2013; Socher et al., 2013; Pei et al., 2014).”
Max-Margin Tensor Neural Network for Chinese Word Segmentation (2014, background): “It was proposed by Hin-ton (1986) and has been a research hot spot for more than twenty years (Bengio et al., 2003; Col-lobert et al., 2011; Schwenk et al., 2012; Mikolov et al., 2013a).”
Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation (2013, background): “Neural LMs have achieved positive results in speech recognition and SMT reranking (Schwenk et al., 2012; Mikolov et al., 2011a).”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, methodology): “Furthermore, RNN models were used to achieve improved word prediction accuracy [83] using the Wall Street Journal benchmark task [84] and to carry out statistical machine translation [85].”