Two decades of statistical language modeling: where do we go from here?

By R. Rosenfeld. Proceedings of the IEEE, 2000

850 citations recorded by the source corpus.

Statistical language models estimate the distribution of various natural language phenomena for the purpose of speech recognition and other language technologies. Since the first significant model was proposed in 1980, many attempts have been made to improve the state of the art. We review them, point to a few promising directions, and argue for a Bayesian approach to integration of linguistic theories with data.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c6586e7c73cc1c9e9a251947425c54c5051be626

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Segmenting DNA sequence into 'words' based on statistical language model (2012, background): “The most common model is n-gram language model (11).”
Factored Language Models and Generalized Parallel Backoff (2003, background): “The ﬁeld is both diverse and intricate (Rosenfeld, 2000; Chen and Goodman, 1998; Jelinek, 1997; Ney et al., 1994), with many different forms of LMs including maximum-entropy, whole-sentence, adaptive and cache-based, to name a small few.”
Offline recognition of unconstrained handwritten texts using HMMs and statistical language models (2004, methodology): “Statistical Language Modeling involves attempts to capture regularities of natural language in order to improve the performance of various natural language applications, e.g., Information Retrieval, Machine Translation, and Document Classification [7].”
Larger-Context Language Modelling with Recurrent Neural Network (2015, background): “The goal of language modelling is to estimate the probability distribution of various linguistic units, e.g., words, sentences (Rosenfeld, 2000).”
Extracting code from programming tutorial videos (2016, methodology): “Our approach is based on two key ideas to effectively extract this code: (i) Leveraging cross-frame information to identify code fragments and improve extraction accuracy, and (ii) Using statistical language models [17] at several levels to capture regularities found in code and use them to improve ”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, background): “Initially proposed by Shannon in 1949 [117], some of the earliest implemented LMs date to the early 1980s and were used as components in systems for automatic speech recognition (ASR), machine translation (MT), document classification, and more [111].”
Toward Deep Learning Software Repositories (2015, background): “Nonetheless, n-grams have limited effective capacity [14].”
Deep learning code fragments for code clone detection (2016, methodology): “To the best of our knowledge, we are the ﬁrst to propose language models and embeddings for clone detection.”
Revisiting the Predictability of Language: Response Completion in Social Media (2012, background): “There has been extensive previous research in language modeling (Rosenfeld, 2000).”
Cluster-based retrieval using language models (2004, background): “A statistical language model is a probability distribution over all possible sentences or other linguistic units in a language [15].”
Statistical language model adaptation: review and perspectives (2004, methodology): “It is therefore necessary to smooth the resulting probability (see e.g., Chen and Rosenfeld, 2000) in order to obtain a robust background SLM.”
Introduction to Digital Speech Processing (2007, background): “The purpose of the language model, or grammar, is to enable the computation of the a priori probability, PL(W ), of a word string, W , consistent with the recognition task [59, 60, 106].”
Exploiting latent semantic information in statistical language modeling (2000, background): “In large part because of this inability to reliably capture large-span behavior, the performance of conventional -gram technology has essentially reached a plateau [61], [72].”
Improving Automatic Source Code Summarization via Deep Reinforcement Learning (2018, background): “Since the number of words coming before a word, y i , varies depending on its location in the input document, p ( y 1 , . . . , y T ) is usually conditioned on a window of n previous words rather than all previous words: This kind of n-grams approach suffers apparent limitations [17], [18].”
Enhancement to the RANKPEP resource for the prediction of peptide binding to MHC molecules using profiles (2004, methodology): “Language models including n-gram models are most frequently applied in speech recognition and natural language tagging (Rosenfeld 2000), but have also been applied to the sequence analysis and motif identification (Jimenez-Montano et al. 2002; Wu and Shivakumar 1994; Wu et al. 1996).”
Automating spoken dialogue management design using machine learning: An industry perspective (2008, background): “Although application developers may be aware of the benefits of statistical modeling, such as greater robustness to out-of-grammar utterances (Rosenfeld, 2000), they often prefer the deterministic character of context-free grammars because they know how to control and modify them.”