Structured Output Layer neural network language model

By Hang Le, Ilya Oparin, Alexander Allauzen, Jean-Luc Gauvain, François Yvon. 2011

157 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Efficient softmax approximation for GPUs (2016, methodology): “In particular, Le et al. (2011) combines a short-list with a hierachical softmax based on word representation.”
Deep Learning: Methods and Applications (2014, background): “Recent published work showed that there is vast room to improve the current optimization techniques for learning deep architectures (Martens, 2010; Le et al., 2011; Martens and Sutskever, 2011; Dean et al., 2012; Sutskever, 2013; Sainath et al., 2013; Wright et al., 2013).”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, background): “One strategy has been to restructure the network to be more hierarchical (Morin and Bengio, 2005; Mnih and Hin-ton, 2009) or to group words into classes (Le et al., 2011).”
Compositional Morphology for Word Representations and Language Modelling (2014, methodology): “Other methods for achieving more drastic complexity reductions exist in the form of frequency-based truncation, shortlists (Schwenk, 2004), or casting the vocabulary as a full hierarchy (Mnih & Hinton, 2008) or partial hierarchy (Le et al., 2011).”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, background): “Moreover, by relaxing the constraint of a binary structure, Le et al. (2011) introduces a structured output layer with an arbitrary tree structure constructed from word clustering.”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “improve the current optimization techniques for learning deep architectures (Martens, 2010; Le et al., 2011; Martens and Sutskever, 2011; Dean et al., 2012; Sutskever, 2013).”
Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation (2012, background): “…the addition of an adaptation layer (Park et al., 2010; Liu et al., 2011) • the so-called SOUL architecture which allows to cover all the words at the output layer instead of using a short-list (Le et al., 2011a; Le et al., 2011b), based on work by (Morin and Bengio, 2005; Mnih and Hinton, 2008).”
Design and Analysis of a Large Corpus of Post-Edited Translations: Quality Estimation, Failure Analysis and the Variability of Post-Edition (2013, background): “Continuous space language models (Le et al., 2011) (features having SOUL in their name) are playing a key role: the importance of most relevant feature, a raw sentence probability estimated by a continuous space language model, is four time larger than the importance of the second most relevant…”
Spoken language understanding using long short-term memory neural networks (2014, background): “In common with feed-forward neural networks [11] [12] [13] [14], an RNN maintains a representation for each word as a high-dimensional real-valued vector.”
Empirical Evaluation and Combination of Advanced Language Modeling Techniques (2011, methodology): “Next, we present results with different neural network based language models, that have been consistently providing very good performance in wide variety of ASR tasks as reported in [3, 18, 4].”
An Overview of Deep-Structured Learning for Information Processing (2011, background): “More recent work on neural network LM with deep architectures can be found in (Le et al., 2010, 2011; Mikolov et al., 2010).”
Linguistic Regularities in Continuous Space Word Representations (2013, methodology): “This has been addressed by subsequent work using hierarchical prediction (Morin and Bengio, 2005; Mnih and Hinton, 2009; Le et al., 2011; Mikolov et al., 2011b; Mikolov et al., 2011a).”
The KIT-LIMSI Translation System for WMT 2015 (2015, background): “Various workarounds have been proposed, relying for instance on a structured output layer using word-classes (Mnih and Hinton, 2008; Le et al., 2011).”
Convolutional Neural Network Language Models (2016, methodology): “Both feed-forward (Schwenk, 2007) and recurrent neural networks (Mikolov et al., 2010) have been shown to outperform n-gram models in various setups (Mikolov et al., 2010; Hai Son et al., 2011).”
LIMSI@WMT’16: Machine Translation of News (2016, background): “…the Structured OUtput Layer (SOUL), it becomes possible to estimate n -gram models that use large output vocabulary, thereby making the training of large neural network language models feasible both for target language models (Le et al., 2011) and translation models (Le et al., 2012a).”
Statistical Language Models Based on Neural Networks (2012, methodology): “Schwenk did use in some cases as little as 2K output units in the neural network, and even if these correspond to the most frequent words, the performance degradation was significant as was later shown in [40].”