Unsupervised Recurrent Neural Network Grammars

By Yoon Kim, Alexander M. Rush, Lei Yu, A. Kuncoro, Chris Dyer, Gábor Melis. 2019

123 citations recorded by the source corpus.

Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but require an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginalizing over the space of latent trees is intractable, we instead apply amortized variational inference. To maximize the evidence lower bound, we develop an inference network parameterized as a neural CRF constituency parser. On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese. On constituency grammar induction, they are competitive with recent neural language models that induce tree structures from words through attention mechanisms.

Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Computational Linguistics & Semantics

Citations in the corpus

Refining Targeted Syntactic Evaluation of Language Models (2021, background): “…(2019) and Wolf (2019) assess the syntactic abilities of large transformers like BERT and GPT2, while Kuncoro et al. (2018), Tran et al. (2018) and Kim et al. (2019) evaluate architectures designed to capture syntax (e.g., Ordered Neurons (Shen et al., 2019) and Recurrent Neural Network Grammars…”
Learning Syntax from Naturally-Occurring Bracketings (2021, methodology): “The models are usually generative and learn from (re)constructing sentences based on induced structures (Shen et al., 2018, 2019; Drozdov et al., 2019; Kim et al., 2019a,b).”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “…for inducing latent syntactic structures, including constituency trees (Choi et al., 2018; Yogatama et al., 2017; Maillard and Clark, 2018; Havrylov et al., 2019; Kim et al., 2019; Drozdov et al., 2019) and dependency trees (Shi et al., 2019), from the distant supervision of downstream tasks.”
Grounded PCFG Induction with Images (2020, methodology): “…so that the lexical expansion model can be tied to the image model (see Section 4.2): (2) where c η is a non-terminal category, W spl ∈ R 2 × h and b spl ∈ R 2 are model parameters for hidden vectors of size h , and x B , c η ∈ R h the result of a multilayered residual network (Kim et al., 2019a).”
Compound Probabilistic Context-Free Grammars for Grammar Induction (2019, methodology): “Additionally, Kim et al. (2019) report perplexity improvements by ﬁne-tuning an RNNG trained on gold trees with the unsupervised RNNG (URNNG)—whereas the RNNG is is trained to maximize the joint likelihood log p ( x , t ) , the URNNG maximizes a lower bound on the log marginal likelihood log t p (…”
Visually Grounded Compound PCFGs (2020, methodology): “The recent advances in the area have been brought by ﬂexible neural models (Jin et al., 2019; Kim et al., 2019a,b; Drozdov et al., 2019).”