LSTM Neural Networks for Language Modeling

By Martin Sundermeyer, Ralf Schlüter, Hermann Ney. 2012

2,027 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, background): “The second LSTM is essentially a recurrent neural network language model [28, 23, 30] except that it is conditioned on the input sequence.”
Character-Level Language Modeling with Deeper Self-Attention (2018, background): “In recent years, strong character-level language models typically follow a common template [32, 31, 40].”
The Curious Case of Neural Text Degeneration (2019, neutral): “Language Model While many neural network architectures have been proposed for language modeling, including LSTMs (Sundermeyer et al., 2012) and convolutional networks (Dauphin et al.”
You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, background): “Modern language models are based on recurrent neural-network architectures [38] such as LSTMs [56] and, more recently, Transformers [17, 47].”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “We report performance of hybrid systems using both a 4-gram count based language model [20] and an LSTM language model [21] in the first pass decoding [22].”
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions (2021, methodology): “DL architectures that demonstrate good fits for NLP include LSTMs [13], RNNs [14], and Transformers [15] that have paved the way for models such as BERT [16], GPT-2 [17], and GPT-3 [18].”
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “Many recent studies have treated neural LMs and contextualized word prediction models—primarily LSTM LMs (Sundermeyer et al., 2012), GPT2 (Radford et al., 2019), and BERT (Devlin et al., 2019)—as psycholinguistic subjects to be studied behaviorally (Linzen et al., 2016; Gulordava et al., 2018; Goldb”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, methodology): “…and Schmidhuber, 1997) or gated recurrent units (GRU) (Cho et al., 2014), has led to significant gains in language modeling (Mikolov et al., 2010; Sundermeyer et al., 2012), parsing (Vinyals et al., 2015; Kiperwasser and Goldberg, 2016; Dyer et al., 2016), machine translation (Bahdanau et al.,…”
Character-Aware Neural Language Models (2015, methodology): “LSTMs have been shown to outperform vanilla RNNs on many tasks, including on language modeling (Sundermeyer, Schluter, and Ney 2012).”
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation (2015, methodology): “Our language model computes log p(wi | w1, . . . , wi−1) by composing representations of words w1, . . . , wi−1 using an recurrent LSTM model (Mikolov et al., 2010; Sundermeyer et al., 2012).”
Learning Longer Memory in Recurrent Neural Networks (2014, neutral): “The LSTM networks were ﬁrst introduced to language modeling by Sundermeyer et al. (2012).”
Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting (2019, background): “Long Short Term Memory (LSTM) [11] networks alleviate that issue and have had great success in langulage modeling and other seq-to-seq tasks [11, 22].”
Deep Learning Stock Volatility with Google Domestic Trends (2015, background): “This particular type of recurrent neural network has shown remarkable results in tasks such as artificial handwriting generation [8], language forecasting [9], and speech recognition [10].”