Sparse Non-negative Matrix Language Modeling

By Noam Shazeer, Joris Pelemans, Ciprian Chelba. 2015

30 citations recorded by the source corpus.

We present Sparse Non-negative Matrix (SNM) estimation, a novel probability estimation technique for language modeling that can efficiently incorporate arbitrary features. We evaluate SNM language models on two corpora: the One Billion Word Benchmark and a subset of the LDC English Gigaword corpus. Results show that SNM language models trained with n-gram features are a close match for the well-established Kneser-Ney models. The addition of skip-gram features yields a model that is in the same league as the state-of-the-art recurrent neural network language models, as well as complementary: combining the two modeling techniques yields the best known result on the One Billion Word Benchmark. On the Gigaword corpus further improvements are observed using features that cross sentence boundaries. The computational advantages of SNM estimation over both maximum entropy and neural network estimation are probably its main strength, promising an approach that has large flexibility in combining arbitrary features and yet scales gracefully to large amounts of data.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “…scale LM has shown that RNNs are great in combination with N-grams, as they may have different strengths that complement N-gram models, but worse when considered in isolation (Mikolov et al., 2011; Mikolov, 2012; Chelba et al., 2013; Williams et al., 2015; Ji et al., 2015a; Shazeer et al., 2015).”
Efficient softmax approximation for GPUs (2016, neutral): “3 Sparse Non-negative Matrix factorization [41] 52.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “7, far surpassed the previous state-ofthe-art ensemble [65], which had a perplexity of 41.”
Frustratingly Short Attention Spans in Neural Language Modeling (2017, background): “Neural language models often work best in combination with traditional N -gram models (Mikolov et al., 2011; Chelba et al., 2013; Williams et al., 2015; Ji et al., 2016; Shazeer et al., 2015), since the former excel at generalization while the latter ensure memorization.”
Neural Network Methods for Natural Language Processing (2017, background): “8Such skip-grams were explored also for non-neural language models; see Pelemans et al. [2016] and the references therein.”
Learning Trans-Dimensional Random Fields with Applications to Language Modeling (2018, methodology): “section mainly include: (1) introducing a novel feature type, the tied long skip-bigram feature “tied” [36], which further demonstrates the flexibility of TRF modeling to use a richer set of features, (2) re-running the experiments with the newly”