Do latent tree learning models identify meaningful structure in sentences?

By Adina Williams, Andrew Drozdov, Samuel R. Bowman. 2017

123 citations recorded by the source corpus.

Recent work on the problem of latent tree learning has made it possible to train neural networks that learn to both parse a sentence and use the resulting parse to interpret the sentence, all without exposure to ground-truth parse trees at training time. Surprisingly, these models often perform better at sentence understanding tasks than models that use parse trees from conventional parsers. This paper aims to investigate what these latent tree learning models learn. We replicate two such models in a shared codebase and find that (i) only one of these models outperforms conventional tree-structured models on sentence classification, (ii) its parsing strategies are not especially consistent across random restarts, (iii) the parses it produces tend to be shallower than standard Penn Treebank (PTB) parses, and (iv) they do not resemble those of PTB or any other semantic or syntactic formalism that the authors are aware of.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “Moreover, MNLI was designed specifically with the goal of being multi-genre in the hope that it would lead to cross-domain learning; indeed, performance of trained models degrades only slightly when tested across domains (Williams et al., 2018b).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “…et al., 2016; Hupkes et al., 2018). computing correlations between neural network activations and some property, for example, correlating RNN state activations with depth in a syntactic tree (Qian et al., 2016a) or with Melfrequency cepstral coefficient (MFCC) acoustic features (Wu and King, 2016).”
Learning Semantic Textual Similarity from Conversations (2018, methodology): “For comparison, we also list the results of Gumbel TreeLSTM (Williams et al., 2017) which is the best sentence encoder based model, and KIM Ensemble (Chen et al., 2017) which is the current state-of-the-art.”
Assessing Composition in Sentence Vector Representations (2018, background): “…relations in sentence representations also relates to work by Linzen et al. (2016), who explore whether LSTMs can learn syntactic dependencies, as well as Williams et al. (2017), who investigate the extent to which parsers that are learned based on a semantic objective produce conventional syntax.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, methodology): “As the next step, we exploit the attention weights of the network to build a graph and induce tree structures for each sentence, showing whether syntactic dependencies between words have been learned or not in the spirit of Williams et al. (2018) and Liu and Lapata (2018).”
Visually Grounded Compound PCFGs (2020, methodology): “We report self F1 (Williams et al., 2018) to show model consistency across runs.”
Hierarchical Transformers for Multi-Document Summarization (2019, background): “Although previous work has shown that similar latent representations are beneficial for downstream NLP tasks (Liu and Lapata, 2018; Kim et al., 2017; Williams et al., 2018; Niculae et al., 2018; Fernandes et al., 2019), much work in multi-document summarization has taken advantage of explicit graph…”
Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders (2019, methodology): “Latent tree models have been shown to perform particularly poorly on attachments at the beginning and end of the sequence (Williams et al., 2018a).”
Unsupervised Recurrent Neural Network Grammars (2019, background): “Williams et al. (2018) observe a similar phenomenon in the context of learning latent trees for classi-ﬁcation tasks.”
Language is Power: Representing States Using Natural Language in Reinforcement Learning. (2019, background): “, 2016] through Syntactic Parsing [Williams et al., 2018] .”
The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models (2018, background): “While these models perform very well on standard datasets and also appear to capture some linguistic structure (Williams et al., 2018; Belinkov et al., 2017; Linzen et al., 2016), they also can be quite brittle, typically”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, background): “, a left-branching or right-branching tree (Williams et al., 2018)), or encounter difficulties in training caused by learning branching policies with Reinforcement Learning (RL) (Yogatama et al.”
Visually Grounded Neural Syntax Acquisition (2019, methodology): “However, most of the methods are not able to produce linguistically sound structures, or even consistent ones with fixed data and hyperparameters but different random initializations (Williams et al., 2018).”
Neural Compositional Denotational Semantics for Question Answering (2018, result): “The trees induced by our model are linguistically plausible, in contrast to prior work on structure learning from semantic objectives (Williams et al., 2018).”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “Other work in this space has tried to understand what models know about syntax [24,28,34,49,54,71,83] or the complex interaction between syntactic and semantic categories [38,78,81,82].”