Deep Neural Network Language Models

By Ebru Arisoy, Tara N. Sainath, Brian Kingsbury, Bhuvana Ramabhadran. 2012

214 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Neural Machine Translation in Linear Time (2016, background): “, 2010) or on feed-forward networks (Bengio et al., 2003; Arisoy et al., 2012), the ByteNet Decoder is based on a novel convolutional structure designed to capture a very long range of past inputs.”
Exploring the Limits of Language Modeling (2016, background): “LMs have played a key role in traditional NLP tasks such as speech recognition (Arisoy et al., 2012), machine translation (Schwenk et al.”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “Language modeling is a basic task in natural language processing, with many applications such as speech recognition (Arisoy et al., 2012) and statistical machine translation (Schwenk et al.”
Deep Learning: Methods and Applications (2014, methodology): “The use of DNNs in the same way for LMs appeared more recently in [8].”
Joint Language and Translation Modeling with Recurrent Neural Networks (2013, methodology): “…which have recently advanced the state of the art in language modeling (Mikolov et al., 2010; Mikolov et al., 2011a; Mikolov, 2012), outperforming multi-layer feed-forward based networks in both perplexity and word error rate in speech recognition (Arisoy et al., 2012; Sundermeyer et al., 2013).”
A Neural Knowledge Language Model (2016, background): “Beyond its usage as a standalone application, it has been an indispensable component in many language/speech tasks such as speech recognition [26, 1], machine translation [17], and dialogue systems [40, 34].”
Strategies for Training Large Vocabulary Neural Language Models (2015, background): “Neural network language models (Bengio et al., 2003; Mikolov et al., 2010) have gained popularity for tasks such as automatic speech recognition (Arisoy et al., 2012) and statistical machine translation (Schwenk et al., 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2014).”
Factored recurrent neural network language model in TED lecture transcription (2012, background): “Recently, neural network based language models (NNLMs), such as the feed-forward NNLM [3, 19], the recurrent NNLM (RNNLM) [15, 16] and the deep NNLM [2], have been continuously reported to perform well amongst other language modeling techniques.”
Automated Crowdturfing Attacks and Defenses in Online Review Systems (2017, background): “In contrast, a generative RNN model can generate much more coherent text [2, 43], but still falls short for larger types of content [50].”
Translation Modeling with Bidirectional Recurrent Neural Networks (2014, background): “In related ﬁelds like e. g. language modeling, this type of neural network has been shown to perform considerably better than standard feedforward architectures (Mikolov et al., 2011; Arisoy et al., 2012; Sundermeyer et al., 2013; Liu et al., 2014).”
Convolutional Neural Network Language Models (2016, background): “These networks have two hidden layers (Arisoy et al., 2012) with the size of 2 times the embedding size ( k ), thus having the same number of parameters as our baseline.”
Lattice rescoring strategies for long short term memory language models in speech recognition (2017, background): “While this makes the N-gram LMs powerful for tasks such as voice-search where short-range contexts sufﬁce, they do not perform as well at tasks such as transcription of long form speech content, that require modeling of long-range contexts [2].”
Recurrent neural networks for language understanding (2013, background): “In recent years, Recurrent Neural Network language models (RNN-LMs) have demonstrated outstanding performance in a variety of natural language processing tasks [1, 2, 3, 4, 5, 6].”
Spoken language understanding using long short-term memory neural networks (2014, background): “In recent years, neural network based approaches have demonstrated outstanding performance in a variety of natural language processing tasks [1] [2] [3] [4] [5] [6] [7] [8].”
Learning Hierarchical Representation Model for NextBasket Recommendation (2015, background): “Besides, one may also consider to introduce nonlinear hidden layers as in deep neural network [4].”
Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation (2013, background): “Both types of neural LMs have seen many improvements recently, in terms of computational scalability (Le et al., 2011) and modeling power (Arisoy et al., 2012; Wu et al., 2012; Alexandrescu and Kirchhoff, 2006).”
From Feedforward to Recurrent LSTM Neural Networks for Language Modeling (2015, methodology): “To the best of our knowledge, so far only [15] investigated the potential gains by deep networks for language modeling.”
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets (2013, methodology): “We apply low-rank factorization to the best scoring DNN architecture in [14], which consists of one projection layer where each word is represented with 120 dimensional features, three hidden layers with 500 hidden units per layer and a softmax layer with 10,000 output targets.”
Automatic speech recognition for under-resourced languages: A survey (2014, methodology): “Artificial Neural Networks (ANN) including single hidden layer NN and multiple hidden layers NN (Deep Neural Networks DNN or Deep Belief Networks DBN) are also used for ASR subtasks such as acoustic modeling [Mohamed, 2012; Seide, 2011] and language modeling [Arisoy, 2012; Mikolov, 2010].”
Lattice decoding and rescoring with long-Span neural network language models (2014, background): “In particular, with existing approaches, the improvements obtained with recurrent neural networks (which were shown to outperform feedforward models in [6, 11, 12]) cannot be transferred to any of the above applications.”