Tree-Structured Composition in Neural Networks without Tree-Structured Architectures

By Samuel R. Bowman, Christopher D. Manning, Christopher Potts. 2015

76 citations recorded by the source corpus.

Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Model Architectures

Citations in the corpus

Structured Attention Networks (2017, methodology): “For this task, we use the Stanford NLI dataset (Bowman et al., 2015) and model our approach off of the decomposable attention model of Parikh et al. (2016).”
Neural Tree Indexers for Text Understanding (2016, background): “Chain-structured LSTM models seem to learn syntactic structure from the natural language however their generalization on unseen text is relatively poor comparing with models that exploit syntactic tree structure (Bowman et al., 2015b).”
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (2019, methodology): “Simpler synthetic languages (not based on nat-ural corpora) have been used in a number of recent studies to examine the inductive biases of different neural architectures (Bowman et al., 2015; Lake and Baroni, 2018; McCoy et al., 2018).”
Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks (2017, unknown): “…with mixed results (e.g., Christiansen & Chater, 1994; Marcus, 1998; Phillips, 1998; Chang, 2002; van der Velde et al., 2004; Botvinick & Plaut, 2006; Wong & Wang, 2007; Bow-ers et al., 2009; Botvinick & Plaut, 2009; Brakel & Frank, 2009; Frank et al., 2009; Frank, 2014; Bowman et al., 2016).”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, methodology): “We also analyze the model’s performance on the logical inference task described in Bowman et al. (2015).”
QCD-aware recursive neural networks for jet physics (2017, result): “A similar observation was already made in the context of natural languages [28{30], where tree-based models have at best only slightly outperformed simpler sequence-based networks.”