Continuous Space Language Models for Statistical Machine Translation

By Holger Schwenk. 2006

187 citations recorded by the source corpus.

Statistical machine translation systems are based on one or more translation models and a language model of the target language. While many different translation models and phrase extraction algorithms have been proposed, a standard word n-gram back-off language model is used in most systems. In this work, we propose to use a new statistical language model that is based on a continuous representation of the words in the vocabulary. A neural network is used to perform the projection and the probability estimation. We consider the translation of European Parliament Speeches. This task is part of an international evaluation organized by the TC-STAR project in 2006. The proposed method achieves consistent improvements in the BLEU score on the development and test data. We also present algorithms to improve the estimation of the language model probabilities when splitting long sentences into shorter chunks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “Traditionally, a neural network trained as a target-side language model has been used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al. , 2006).”
Recurrent Continuous Translation Models (2013, methodology): “Phrase-based continuous translation models were ﬁrst proposed in (Schwenk et al., 2006) and re-1700 cently further developed in (Schwenk, 2012; Le et al., 2012).”
Fast and Robust Neural Network Joint Models for Statistical Machine Translation (2014, methodology): “In recent years, neural network models have become increasingly popular in NLP. Initially, these models were primarily used to create n -gram neural network language models (NNLMs) for speech recognition and machine translation (Bengio et al., 2003; Schwenk, 2010).”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, methodology): “NPLMs have previously been applied to MT, most notably feed-forward NPLMs (Schwenk, 2007; Schwenk, 2010) and RNN-LMs (Mikolov, 2012).”
Compositional Morphology for Word Representations and Language Modelling (2014, methodology): “Rescoring lattices with a CSLM proved to be beneﬁcial for ASR (Schwenk, 2004) and was subsequently applied to translation (Schwenk et al., 2006; Schwenk & Koehn, 2008), reaching training sizes of up to 500m words (Schwenk et al., 2012).”
Pragmatic Neural Language Modelling in Machine Translation (2014, methodology): “…set of words over which the normalisation is performed, either via class-based factorisation (Botha and Blunsom, 2014; Baltescu et al., 2014) or using a shortlist containing only the most frequent words in the vocabulary and scoring the remaining words with a back-off n-gram model (Schwenk, 2010).”
Continuous Space Translation Models for Phrase-Based Statistical Machine Translation (2012, methodology): “Our implementation is based on the open-source CSLM toolkit described in (Schwenk, 2010; Schwenk et al., 2012).”
Neural Machine Translation (2020, unknown): “They made their implementation available as a open source toolkit (Schwenk, 2010), which also supports training on a graphical processing unit (GPU) (Schwenk et al.”
Recurrent neural networks for language understanding (2013, background): “In conjunction with training on large datasets, continuous space language models have resulted in improvements in language model perplexity [3, 9, 8] machine translation Bilingual Evaluation Understudy (BLEU) score [14, 15, 16] and speech recognition Word Error Rate (WER) [7, 10, 6].”
Continuous space language models (2007, unknown): “Statistical machine translation is not considered in this work, but initial studies have shown that the proposed neural network language model can also be used for this task (Schwenk et al., 2006).”
Neural Machine Translation (2016, unknown): “FFNLM can be integrated as a feature in the log-linear SMT model [Schwenk et al., 2006] costly due to matrix multiplications and softmax solutions: n-best reranking variants of softmax (hierarchical softmax, self-normalization [NCE]) shallow networks; premultiplication of hidden layer scale well to ”
Training Continuous Space Language Models: Some Practical Issues (2010, methodology): “This approach has showed signiﬁcant and consistent improvements when applied to automatic speech recognition (Schwenk, 2007; Emami and Mangu, 2007; Kuo et al., 2010) and machine translation tasks (Schwenk et al., 2006).”
LIMSI@WMT’15 : Translation Task (2015, unknown): “Neural networks, working on top of conventional n -gram back-off language models, have been introduced in (Bengio et al., 2003; Schwenk et al., 2006) as a potential means to improve conventional language models.”
Letter N-Gram-based Input Encoding for Continuous Space Language Models (2013, methodology): “In Schwenk et al. (2006) they use the CSLM framework to rescore n-best lists of a machine translation system during tuning and testing steps.”
Neural Machine Translation: A Review (2019, background): “, 2003, 2006) for the target language to rerank translation lattices (Schwenk et al., 2006).”
Translation Modeling with Bidirectional Recurrent Neural Networks (2014, background): “Several groups have reported strong improvements over state-of-the-art baselines using feedforward neural network-based language models (Schwenk et al., 2006; Vaswani et al., 2013), as well as translation models (Le et al., 2012; Schwenk, 2012; Devlin et al., 2014).”