Continuous space language models

By Holger Schwenk. 2007

604 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “To address the computational complexity of using a CSLM in the decoder a buffer was used to aggregate n-grams during the stack-search performed by the decoder.”
Distributed Representations of Words and Phrases and their Compositionality (2013, background): “The follow up work includes applications to automatic speech recognit ion and machine translation [14, 7], and a wide range of NLP tasks [2, 20, 15, 3, 18, 19, 9].”
Efficient Estimation of Word Representations in Vector Space (2013, background): “For example, neural network based language models significantly outperform N-gram models [1, 25, 16].”
On Using Monolingual Corpora in Neural Machine Translation (2015, methodology): “…the application of neural networks to machine translation Equal contribution † Work done while author was at Universit ´ e de Montr ´ eal was restricted to extend standard machine translation tools for rescoring translation hypotheses or reranking n-best lists (see, e.g., Schwenk, 2012, 2007a).”
Efficient softmax approximation for GPUs (2016, background): “This is achieved by explicitly taking into account the computational complexity of parallel systems and combining it with a few important observations, namely keeping a short-list of frequent words in the root node [39] and reducing the capacity of rare words [8].”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “Neural networks have been shown to work well with word tokens (Bengio et al., 2001; Schwenk, 2007; Mikolov et al., 2010) but also with ﬁner units, such as subwords (Sen-nrich et al., 2015; Botha and Blunsom, 2014; Lu-ong et al., 2013) as well as symbols resulting from compression/encoding (Chitnis…”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, methodology): “NPLMs have previously been applied to MT, most notably feed-forward NPLMs (Schwenk, 2007; Schwenk, 2010) and RNN-LMs (Mikolov, 2012).”
Larger-Context Language Modelling with Recurrent Neural Network (2015, methodology): “s to language modelling have been extensively tested against each other in terms of speech recognition and machine translation in recent years (Sundermeyer et al., 2015; Baltescu & Blunsom, 2014; Schwenk, 2007). Often the conclusion is that all three techniques tend to have different properties ”
The IBM 2015 English conversational telephone speech recognition system (2015, methodology): “For LM rescoring, we used two types of LMs: model M, a class-based exponential model [25] and neural network LM (NNLM) [26, 27, 28, 29].”
LSTM Neural Networks for Language Modeling (2012, background): “In addition, we gain considerable improvements in WER on top of a state-of-the-art speech recognition system.”
Incremental Adaptation Strategies for Neural Network Language Models (2014, background): “Various solutions have been proposed: short-lists (Schwenk, 2007), a class decomposition (Mikolov et al.”
From Feedforward to Recurrent LSTM Neural Networks for Language Modeling (2015, background): “In particular, previous work has concentrated on feedforward neural networks ([7], [8]), recurrent neural networks ([9]), and recurrent long short-term memory (LSTM) neural networks ([10]).”
Applying deep learning to answer selection: A study and an open task (2015, background): “on image recognition [9], on speech [10] [11] [12] and on machine translation [13] [14].”
Generalizing and Hybridizing Count-based and Neural Language Models (2016, background): “Feed-forward neural network LMs: Feedforward LMs (Bengio et al., 2006; Schwenk, 2007) are LMs that, like n-grams, calculate the probability of the next word based on the previous words.”
Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation (2013, background): “1 Another major type of neural LMs are the so-called feed-forward networks (Bengio et al., 2003; Schwenk, 2007; Nakamura et al., 1990).”
Recurrent neural networks for language understanding (2013, background): “In recent years, Recurrent Neural Network language models (RNN-LMs) have demonstrated outstanding performance in a variety of natural language processing tasks [1, 2, 3, 4, 5, 6].”