A Neural Syntactic Language Model

By Ahmad Emami, Frederick Jelinek. 2005

79 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Recurrent Neural Network Grammars (2016, background): “There is an extensive literature here, but one strand of work has emphasized a bottom-up generation of the tree, using variants of shift-reduce parser actions to deﬁne the probability space (Chelba and Jelinek, 2000; Emami and Jelinek, 2005).”
A Fast Unified Model for Parsing and Sentence Understanding (2016, background): “There is a fairly long history of work on building neural network-based parsers that use the core operations and data structures from transition-based parsing, of which shift-reduce parsing is a variant (Henderson, 2004; Emami and Jelinek, 2005; Titov and Henderson, 2010; Chen and Manning, 2014; Buy”
Joint Language and Translation Modeling with Recurrent Neural Networks (2013, methodology): “A well established efﬁciency trick uses word-classing to create a more efﬁcient two-step process (Goodman, 2001; Emami and Jelinek, 2005; Mikolov et al., 2011b) where each word is assigned a unique class.”
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, neutral): “As RNNGs build phrase-structure trees through top-down operations, we propose extensions to the structure-building sequences and model architecture that enable left-corner (Henderson, 2003, 2004) and bottom-up (Chelba and Jelinek, 2000; Emami and Jelinek, 2005) generation orders (§4).”
Blank Language Models (2020, methodology): “Part of these efforts was focused on ﬁnding an optimal generation order, including syntax-based approaches and methods for learning adaptive generation order (Emami & Jelinek, 2005; Zhang et al., 2015; Dyer et al., 2016; Ford et al., 2018; Zhou et al., 2019; Welleck et al., 2019; Gu et al., 2019a).”
A Bayesian Model for Generative Transition-based Dependency Parsing (2015, background): “2001; Charniak, 2001) show improvements over n-gram models, but decoding is prohibitively expensive for practical integration in language generation applications. Chelba and Jelinek (2000) as well as Emami and Jelinek (2005) proposed incremental syntactic language models with some similarities to ou”
Non-Monotonic Sequential Text Generation (2019, background): “For example, Emami & Jelinek (2005) proposed to factorize the probability over a sentence following its syntactic structure and train a neural network to model conditional distributions, which was followed more recently by Zhang et al. (2015) and by Dyer et al. (2016).”
Structured Output Layer Neural Network Language Models for Speech Recognition (2013, background): “These methods are also appealing, as the WordNet or similar resources covering large output vocabularies are not necessarily available for different languages.”
Generative Incremental Dependency Parsing with Neural Networks (2015, methodology): “Syntactic language models based on PCFGs (Roark, 2001; Charniak, 2001) and incremental parsing (Chelba and Jelinek, 2000; Emami and Jelinek, 2005) have been proposed for speech recognition and machine translation.”
The IBM 2015 English conversational telephone speech recognition system (2015, methodology): “For LM rescoring, we used two types of LMs: model M, a class-based exponential model [25] and neural network LM (NNLM) [26, 27, 28, 29].”
Sparse Non-negative Matrix Language Modeling (2015, background): “Recently, neural network (NN) smoothing [1], [5], [18], and in particular recurrent neural networks (RNNs) [12], [20] have shown excellent performance in language modeling [3].”
Unsupervised Recurrent Neural Network Grammars (2019, background): “There has been much work on incorporating tree structures into deep models for syntax-aware language modeling, both for unconditional (Emami and Jelinek, 2005; Buys and Blunsom, 2015; Dyer et al., 2016) and conditional (Yin and Neubig, 2017; Alvarez-Melis and Jaakkola, 2017; Rabi-novich et al.,…”
Extensions of recurrent neural network language model (2011, methodology): “132 Structured LM (Filimonov) [9] 125 Syntactic NN LM (Emami) [10] 107 RNN trained by BP 113 RNN trained by BPTT 106 4x RNN trained by BPTT (mixture) 98 where f (z) and g(z) are sigmoid and softmax activation functions (the softmax function in the output layer is used to make sure that the outputs f”
Deep Neural Network Language Models (2012, methodology): “Since the input structure of NNLM allows for using larger contexts with a little complexity, NNLM was also investigated in syntactic-based language modeling to efﬁciently use long distance syntactic information (Emami, 2006; Kuo et al., 2009).”