Compound Probabilistic Context-Free Grammars for Grammar Induction

By Yoon Kim, Chris Dyer, Alexander M. Rush. 2019

167 citations recorded by the source corpus.

We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar. In contrast to traditional formulations which learn a single stochastic grammar, our context-free rule probabilities are modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions. Inference in this context-dependent grammar is performed by collapsed variational inference, in which an amortized variational posterior is placed on the continuous variable, and the latent trees are marginalized with dynamic programming. Experiments on English and Chinese show the effectiveness of our approach compared to recent state-of-the-art methods for grammar induction from words with neural language models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Grammar-Based Grounded Lexicon Learning (2022, methodology): “However, our model is fundamentally different from works on unsupervised constituency parsing [25, 36] which use the CKY algorithm for inference over scalar span scores and those compute span representation vectors with CKY-style algorithms [29, 12, inter alia ].”
Visually Grounded Compound PCFGs (2020, methodology): “The recent advances in the area have been brought by ﬂexible neural models (Jin et al., 2019; Kim et al., 2019a,b; Drozdov et al., 2019).”
Grounded PCFG Induction with Images (2020, methodology): “(2) where cη is a non-terminal category, Wspl ∈ R2×h and bspl ∈ R2 are model parameters for hidden vectors of size h, and xB,cη ∈ Rh the result of a multilayered residual network (Kim et al., 2019a).”
Learning Syntax from Naturally-Occurring Bracketings (2021, methodology): “The models are usually generative and learn from (re)constructing sentences based on induced structures (Shen et al., 2018, 2019; Drozdov et al., 2019; Kim et al., 2019a,b).”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “Our PCFG uses the neural parameterization from Kim et al. [64].”
Leveraging Language to Learn Program Abstractions and Search Heuristics (2021, background): “A promising future direction could leverage recent structured, neural joint models that can learn the compositional units of language, and incorporate pre-trained language representations (Joshi & Schabes, 1997; Wiseman et al., 2018; Kim et al., 2019).”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, methodology): “Grammar based methods have been used for a variety of tasks such as program translation [9], conditional program generation [71,72], grammar induction [32] and generative modelling on structures with syntax [37,12], such as molecules.”