From Feedforward to Recurrent LSTM Neural Networks for Language Modeling

By Martin Sundermeyer, Hermann Ney, Ralf Schlüter. 2015

481 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Representation learning for very short texts using weighted word embedding aggregation (2016, background): “Recurrent neural networks (RNNs) and variants can overcome the problem of ﬁxed dimensionality or aggregation, since one can feed word after word in the system and in the end arrive at a text representation (Sutskever et al. (2014); Sor-doni et al. (2015); Sundermeyer et al. (2015)).”
Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers (2016, background): “This was motivated by recent successes of applying recurrent networks to natural languages (see, e.g., (Cho et al., 2014; Sundermeyer et al., 2015)) and from the fact that the recurrent network can efﬁciently capture long-term dependencies even with a single layer.”
Larger-Context Language Modelling with Recurrent Neural Network (2015, methodology): “These different approaches to language modelling have been extensively tested against each other in terms of speech recognition and machine translation in recent years (Sundermeyer et al., 2015; Baltescu & Blunsom, 2014; Schwenk, 2007).”
Auto-Sizing Neural Networks: With Applications to n-gram Language Models (2015, methodology): “We evaluate our model using the open-source NPLM toolkit released by Vaswani et al. (2013), extending it to use the additional regularizers as described in this paper.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “Generating multiple predictions is straightforward for traditional left-to-right LMs (Sundermeyer et al., 2015; Radford et al., 2019), where we can autoregressively decode the next token conditioned on previous tokens.”
A Neural Approach to Automated Essay Scoring (2016, background): “Apart from these examples, recurrent neural networks have also been used for opinion mining (Ir-soy and Cardie, 2014), sequence labeling (Ma and Hovy, 2016), language modeling (Kim et al., 2016; Sundermeyer et al., 2015), etc.”
End-to-End Speech Recognition: A Survey (2023, background): “Also, the language model training objective, i.e. language model perplexity, is observed to correlate well with word error rate [311], [312], [313], [314].”
Automated Atrial Fibrillation Detection using a Hybrid CNN-LSTM Network on Imbalanced ECG Datasets (2021, methodology): “Besides CNN, Long Short-Term Memory (LSTM) networks [17] are another type of DL techniques frequently used for time series analysis [18], including speech recognition [19] and language processing [20].”
Spectral-Spatial Attention Networks for Hyperspectral Image Classification (2019, background): “RNN, which extends conventional feedforward neural networks with loops in connections, has gained signiﬁcant attention for solving many challenging problems involving sequential data analysis, such as speech recognition and language modeling [31,32].”
Lattice rescoring strategies for long short term memory language models in speech recognition (2017, methodology): “To reduce the computation involved in integrating an LSTMLM into the first pass of a recognizer [6, 7], we generate lattices using an initial recognition pass that incorporates the acoustic model and the Ngram LM.”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, methodology): “[87] compared the gain obtained by replacing a feed-forward neural network with an RNN when conditioning the prediction of a word on the words ahead.”
A preliminary study on deep learning for predicting social insurance payment behavior (2017, methodology): “Hence, predicting the payment status of an insured covered by National Pension Insurance can be addressed as sequence-to-sequence prediction model [9] [19] [21] [22].”
The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation (2016, methodology): “For the RASR back end (R) we carried out lattice rescoring [20] with an LSTM-RNN language model [21, 22] as follows.”