Pragmatic Neural Language Modelling in Machine Translation

By Paul Baltescu, Phil Blunsom. 2014

34 citations recorded by the source corpus.

This paper presents an in-depth investigation on integrating neural language models in translation systems. Scaling neural language models is a difficult task, but crucial for real-world applications. This paper evaluates the impact on end-to-end MT quality of both new and existing scaling techniques. We show when explicitly normalising neural models is necessary and what optimisation tricks one should use in such scenarios. We also focus on scalable training algorithms and investigate noise contrastive estimation and diagonal contexts as sources for further speed improvements. We explore the trade-offs between neural models and back-off n-gram models and find that neural models make strong candidates for natural language applications in memory constrained environments, yet still lag behind traditional models in raw translation quality. We conclude with a set of recommendations one should follow to build a scalable neural language model for MT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Adaptive Input Representations for Neural Language Modeling (2018, methodology): “Language modeling is a basic task in natural language processing, with many applications such as speech recognition (Arisoy et al., 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu & Blunsom, 2015).”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “NCE and it variants has gained popularity in natural language processing (NLP) [23, 57, 2, 4].”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “It has a target vocabulary of 100K and is trained using noise-contrastive estimation (Mnih and Teh, 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2015; Williams et al., 2015).”
Larger-Context Language Modelling with Recurrent Neural Network (2015, methodology): “These different approaches to language modelling have been extensively tested against each other in terms of speech recognition and machine translation in recent years (Sundermeyer et al., 2015; Baltescu & Blunsom, 2014; Schwenk, 2007).”
Strategies for Training Large Vocabulary Neural Language Models (2015, background): “Neural network language models (Bengio et al., 2003; Mikolov et al., 2010) have gained popularity for tasks such as automatic speech recognition (Arisoy et al., 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2014).”
Dynamic Entity Representations in Neural Language Models (2017, background): “CFSM is a class factorized softmax function (Goodman, 2001; Baltescu and Blunsom, 2015).”
Neural Machine Translation (2020, unknown): “Baltescu and Blunsom (2015) compare the two techniques class-based word encoding with normalized scores vs. noise-contrastive estimation without normalized scores - and show that the letter gives better performance with much higher speed. As another way to allow straightforward decoder integration, ”
Generative Incremental Dependency Parsing with Neural Networks (2015, background): “Classbased factorization has been shown to be an effective strategy in normalizing neural language models (Baltescu and Blunsom, 2015), The class prediction score is defined as ψ(c,h) = sc φ(h) + dc, where sc ∈ RD is the output weight vector for class c and dc is the class bias weight.”