SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS

By Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hang Le, Stefan Kombrink, Jan Honza Cernocký. 2011

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Neural Machine Translation of Rare Words with Subword Units (2015, methodology): “Mikolov et al. (2012) investigate subword language models, and propose to use syllables.”
Generating Sequences With Recurrent Neural Networks (2013, methodology): “Character-level language modelling with neural networks has recently been considered [30, 24], and found to give slightly worse performance than equivalent word-level models.”
Enriching Word Vectors with Subword Information (2016, methodology): “A first class of such models are recurrent neural networks, applied to language modeling (Mikolov et al., 2012; Sutskever et al., 2011; Graves, 2013; Bojanowski et al., 2015), text normalization (Chrupała, 2014), part-of-speech tagging (Ling et al.”
HyperNetworks (2016, methodology): “In the following experiments, we will benchmark the performance of static hypernetworks on image recognition with MNIST and CIFAR-10, and the performance of dynamic hypernetworks on language modelling with Penn Treebank and Hutter Prize Wikipedia (enwik8) datasets and handwriting generation.”
FastText.zip: Compressing text classification models (2016, background): “Additionally, instead of pruning out the less useful features, we can decompose them into smaller units (Mikolov et al., 2012).”
On the difficulty of training recurrent neural networks (2012, methodology): “We address the task of polyphonic music prediction, using the datasets Piano-midi.de, Nottingham and MuseData described in Boulanger-Lewandowski et al. (2012) and language modelling at the character level on the Penn Treebank dataset (Mikolov et al. , 2012).”
Character-Aware Neural Language Models (2015, background): “However they are generally outperformed by word-level models (Mikolov et al. 2012).”
Gated Feedback Recurrent Neural Networks (2015, methodology): “Closely following the protocols in (Mikolov et al., 2012; Graves, 2013), we used the first 90 MBytes of characters to train a model, the next 5 MBytes as a validation set, and the remaining as a test set, with the vocabulary of 205 characters including a token for an unknown character.”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “Although there have been a number of previous research reporting the use of neural networks with characters (see, e.g., Mikolov et al. (2012) and Santos and Zadrozny (2014)), the dominant approach has been to preprocess the text into a sequence of symbols, each associated with a sequence of…”
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019, background): “Compared to natural language, amino acid sequences are also much longer and use a vocabulary of only 25 elements, and are thus more similar to character-level language models (Mikolov et al., 2012; Kim et al., 2016) than traditional word-level models.”
Secure Evaluation of Quantized Neural Networks (2019, background): “The effect of this approach, which can be already seen as some form of quantization, has not been extensively tested, not even in the reference [39] provided by the authors on this matter.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “We also used the text8 dataset for character-level language modeling (Mikolov et al., 2012). text8 is about 20 times larger than PTB, with about 100M characters from Wikipedia (90M for training, 5M for validation, and 5M for testing).”
On Multiplicative Integration with Recurrent Neural Networks (2016, methodology): “All models have a single hidden layer of size 2048, and we use Adam optimization algorithm [13] with learning rate 1e−4.”
Dynamic Evaluation of Neural Sequence Models (2017, neutral): “architecture test set error(bits/char) mRNN (Mikolov et al., 2012) 1.”
Multiplicative LSTM for sequence modelling (2016, methodology): “architecture test set error (bits/char) mRNN (Mikolov et al., 2012) 1.”