Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders

By Andrew Drozdov, Pat Verga, Mohit Yadav, Mohit Iyyer, Andrew McCallum. 2019

129 citations recorded by the source corpus.

We introduce the deep inside-outside recursive autoencoder (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree. Our approach predicts each word in an input sentence conditioned on the rest of the sentence. During training we use dynamic programming to consider all possible binary trees over the sentence, and for inference we use the CKY algorithm to extract the highest scoring parse. DIORA outperforms previously reported results for unsupervised binary constituency parsing on the benchmark WSJ dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Yoav Goldberg
3. Christopher D. Manning
4. Noah A. Smith
5. Graham Neubig
6. Luke S. Zettlemoyer
7. Chris Dyer
8. Ivan Titov

In Natural Language Processing

1. Mohit Bansal
2. Chris Dyer
3. Alexander M. Rush
4. Ivan Titov
5. Graham Neubig
6. Samuel R. Bowman
7. Kyunghyun Cho
8. Yoav Goldberg

In Model Architectures

1. Chris Dyer
2. Yoon Kim
3. Alexander M. Rush
4. Roger Levy
5. Jiayuan Mao
6. Ivan Titov
7. Lillian Lee
8. Josh Tenenbaum

Citations in the corpus

• Grammar-Based Grounded Lexicon Learning (2022, methodology): “However, our model is fundamentally different from works on unsupervised constituency parsing [25, 36] which use the CKY algorithm for inference over scalar span scores and those compute span representation vectors with CKY-style algorithms [29, 12, inter alia ].”
• Learning Syntax from Naturally-Occurring Bracketings (2021, background): “The models are usually generative and learn from (re)constructing sentences based on induced structures (Shen et al., 2018, 2019; Drozdov et al., 2019; Kim et al., 2019a,b).”
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction (2020, methodology): “Other studies include Drozdov et al. (2019), who trained deep inside-outside recursive autoencoders (DIORA) to derive syntactic trees in an exhaustive way with the aid of the inside-outside algorithm, and Kim et al. (2019a) who proposed Compound Probabilistic Context-Free Grammars (compound PCFG),…”
• Visually Grounded Neural Syntax Acquisition (2019, unknown): “…for inducing latent syntactic structures, including constituency trees (Choi et al., 2018; Yogatama et al., 2017; Maillard and Clark, 2018; Havrylov et al., 2019; Kim et al., 2019; Drozdov et al., 2019) and dependency trees (Shi et al., 2019), from the distant supervision of downstream tasks.”
• Compound Probabilistic Context-Free Grammars for Grammar Induction (2019, methodology): “On standard benchmarks for English and Chinese, the proposed approach is found to perform favorably against recent neural network-based approaches to grammar induction (Shen et al., 2018, 2019; Drozdov et al., 2019; Kim et al., 2019).”
• Visually Grounded Compound PCFGs (2020, methodology): “The recent advances in the area have been brought by flexible neural models (Jin et al., 2019; Kim et al., 2019a,b; Drozdov et al., 2019).”