N-gram Language Modeling using Recurrent Neural Network Estimation

By Ciprian Chelba, Mohammad Norouzi, Samy Bengio. 2017

50 citations recorded by the source corpus.

We investigate the effective memory depth of RNN models by using them for $n$-gram language model (LM) smoothing. Experiments on a small corpus (UPenn Treebank, one million words of training data and 10k vocabulary) have found the LSTM cell with dropout to be the best model for encoding the $n$-gram state when compared with feed-forward and vanilla RNN models. When preserving the sentence independence assumption the LSTM $n$-gram matches the LSTM LM performance for $n=9$ and slightly outperforms it for $n=13$. When allowing dependencies across sentence boundaries, the LSTM $13$-gram almost matches the perplexity of the unlimited history LSTM LM. LSTM $n$-gram smoothing also has the desirable property of improving with increasing $n$-gram order, unlike the Katz or Kneser-Ney back-off estimators. Using multinomial distributions as targets in training instead of the usual one-hot target is only slightly beneficial for low $n$-gram orders. Experiments on the One Billion Words benchmark show that the results hold at larger scale: while LSTM smoothing for short $n$-gram contexts does not provide significant advantages over classic N-gram models, it becomes effective with long contexts ($n > 5$); depending on the task and amount of data it can match fully recurrent LSTM models at about $n=13$. This may have implications when modeling short-format text, e.g. voice search/query LMs. Building LSTM $n$-gram LMs may be appealing for some practical situations: the state in a $n$-gram LM can be succinctly represented with $(n-1)*4$ bytes storing the identity of the words in the context and batches of $n$-gram contexts can be processed in parallel. On the downside, the $n$-gram context encoding computed by the LSTM is discarded, making the model more expensive than a regular recurrent LSTM LM.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, methodology): “Recently, Chelba et al. (2017), in proposing a new model, showed that on PTB, an LSTM language model with 13 tokens of context is similar to the inﬁnite-context LSTM performance, with close to an 8% 5 increase in perplexity.”
Limits of Detecting Text Generated by Large-Scale Language Models (2020, background): “Empirical perplexity comparisons show that LSTM and similar neural language models have Markov order as small as k = 13 [32].”
Colorless Green Recurrent Networks Dream Hierarchically (2018, methodology): “…(singular or plural); second, a 5-gram model with Kneser-Ney smoothing ( KN , Kneser and Ney, 1995) trained using the IRSTLM package (Federico et al., 2008) and queried us-ing KenLM (Heaﬁeld, 2011); and third, a 5-gram LSTM , which only had access to windows of ﬁve tokens (Chelba et al., 2017).”
Advancing Sequence-to-Sequence Based Speech Recognition (2019, background): “Nevertheless, the usual comparisons consider hybrid results after decoding with count based language model, and often discard the fact that encoder-decoder models learn neuralnetwork-like word-sequence representation internally, which is much more powerful than count based models [14, 15].”
What’s Going On in Neural Constituency Parsers? An Analysis (2018, result): “Previously, Chelba et al. (2017) had similar ﬁndings in language modeling, where us-ing LSTMs truncated to a particular distance improved performance over feedforward networks that were given the same context.”