Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation
By Noam Shazeer, Joris Pelemans, Ciprian Chelba. 2014
14 citations recorded by the source corpus.
We present a novel family of language model (LM) estimation techniques named Sparse Non-negative Matrix (SNM) estimation. A first set of experiments empirically evaluating it on the On e Billion Word Benchmark [Chelba et al., 2013] shows that SNM n-gram LMs perform almost as well as the well-established Kneser-Ney (KN) models. When using skip-gram features the models are able to match the state-of-the-art recurrent neural network (RNN) LMs; combining the two modeling techniques yields the best known result on the benchmark. The computational advantages of SNM over both maximum entropy and RNN LM estimation are probably its main strength, promising an approach that has the same flexibility in combining arbitrary feature s effectively and yet should scale to very large amounts of data as gracefully as n-gram LMs do.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Michael Auli
- Angela Fan
- Luke S. Zettlemoyer
- Quoc V. Le
- Mohit Bansal
- Kyunghyun Cho
- David Grangier
- Jason Aaron Edward Weston
Citations in the corpus
- Language Modeling with Gated Convolutional Networks (2016, background): “6 100 CPUs Sparse Non-Negative Matrix LM (Shazeer et al., 2014) 52.”
- Sparse Non-negative Matrix Language Modeling (2015, methodology): “The interested reader can find these in [15].”