Connectionist language modeling for large vocabulary continuous speech recognition

By Holger Schwenk, Jean-Luc Gauvain. 2002

166 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Natural Language Processing (Almost) from Scratch (2011, methodology): “System Accuracy Shen et al. (2007) 97.”
Speed regularization and optimality in word classing (2013, methodology): “Word classing has since been incorporated in several computationally complex language models: Model M [5, 6] where it achieves both likelihood and speed improvements; and in feedforward and recurrent neural networks [7, 8, 9, 10, 11].”
Improving Word Representations via Global Context and Multiple Word Prototypes (2012, background): “Neural language models (Bengio et al., 2003; Mnih and Hinton, 2007; Collobert and Weston, 2008; Schwenk and Gauvain, 2002; Emami et al., 2003) have been shown to be very powerful at language modeling, a task where models are asked to accurately predict the next word given previously seen words.”
Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation (2012, background): “It is usually used to rescore lattices and improvements of the word error rate of about one point were consistently observed for many languages and domains, for instance (Schwenk and Gauvain, 2002; Schwenk, 2007; Park et al., 2010; Liu et al., 2011; Lamel et al., 2011).”
Word Representations: A Simple and General Method for Semi-Supervised Learning (2010, background): “Neural language models (Bengio et al., 2001; Schwenk & Gauvain, 2002; Mnih & Hinton, 2007; Collobert & Weston, 2008), on the other hand, induce dense real-valued low-dimensional 384 word embeddings using unsupervised approaches.”
Hierarchical Probabilistic Neural Network Language Model (2005, methodology): “In ( Schwenk and Gauvain, 2002; Schwenk, 2004), it is shown how the model can be used to directly improve speech recognition performance.”
A unified architecture for natural language processing: deep neural networks with multitask learning (2008, unknown): “Previous Work on Language Models (Bengio & Ducharme, 2001) and (Schwenk & Gauvain, 2002) already presented very similar language models.”
Structured Output Layer Neural Network Language Models for Speech Recognition (2013, background): “The scaling factor depends on the history and was defined in [5] as: (2) As both and are normalized to sum to one, the “combined” conditional distribution is also normalized.”
A Neural Syntactic Language Model (2005, background): “For example in word error rate (WER) experiments the output vocabulary can be limited to a certain number of most frequent words, which would be a fraction of the actual vocabulary ( Schwenk & Gauvain, 2002 ).”
A Challenge Set for Advancing Language Modeling (2012, methodology): “At the word level, N-gram modeling techniques have been extensively refined, with stateof-the-art techniques based on smoothed N-gram counts (Kneser and Ney, 1995; Chen and Goodman, 1999), multi-layer perceptrons (Schwenk and Gauvain, 2002; Schwenk, 2007) and maximumentropy models (Rosenfeld, 1997; ”
Curriculum learning (2009, methodology): “Architecture The architecture of our language model (Figure 4) follows the work introduced by Bengio et al. (2001) and Schwenk and Gauvain (2002), and closely resembles the one used in Collobert and Weston (2008).”
Translation Quality Estimation using Recurrent Neural Network (2016, methodology): “The RNN-LM represents each word as high-dimensional real-valued vectors, like the other continuous space language models such as feed forward neural network language models (Schwenk and Gauvain, 2002; Bengio et al., 2003; Morin and Bengio, 2005; Schwenk, 2007) and Hierarchical Log-Bi-linear language”
CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences (2022, background): “Note that traditional statistical LMs were not able to achieve this level of generalization [47].”
Training Continuous Space Language Models: Some Practical Issues (2010, background): “Training these models for large scale tasks is therefore challenging, and a number of tricks have been introduced to make training and inference tractable (Schwenk and Gauvain, 2002; Schwenk, 2007).”
Random forests and the data sparseness problem in language modeling (2007, background): “Neural network language models and discriminative language models can achieve superior performance in reducing the perplexity and word error rate (Bengio et al., 2001; Schwenk and Gauvain, 2002; Emami et al., 2003; Roark et al., 2004a,b).”
Learning Deep Architectures for AI (2007, methodology): “Such conditional energy-based models have also been exploited in a series of probabilistic language models based on neural networks [15, 16, 130, 169, 170, 171, 207].”