Learning to Compose Task-Specific Tree Structures

By Jihun Choi, Kang Min Yoo, Sang-goo Lee. 2017

196 citations recorded by the source corpus.

For years, recursive neural networks (RvNNs) have been shown to be suitable for representing text into fixed-length vectors and achieved good performance on several natural language processing tasks. However, the main drawback of RvNNs is that they require structured input, which makes data preparation and model implementation hard. In this paper, we propose Gumbel Tree-LSTM, a novel tree-structured long short-term memory architecture that learns how to compose task-specific tree structures only from plain text data efficiently. Our model uses Straight-Through Gumbel-Softmax estimator to decide the parent node among candidates dynamically and to calculate gradients of the discrete decision. We evaluate the proposed model on natural language inference and sentiment analysis, and show that our model outperforms or is at least comparable to previous models. We also find that our model converges significantly faster than other models.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017), but recent non-attentional models designed to extract useful sentence representations have nearly closed the gap (Liu et al., 2017b; Im and Cho, 2017; Shen et al., 2018; Choi et al., 2017).”
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information (2018, background): “The first type of these approaches is sentence-encoding-based methods [7, 6, 24, 34] where sentences are encoded into their own sentence representation without any cross-interaction.”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “, 2020), machine translation (Su et al., 2020; Choi et al., 2017; Eriguchi et al., 2016), dialogue (Rao et al.”
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling (2018, unknown): “5 600D Gumbel TreeLSTM encoders [Choi et al., 2017b] 10m 93.”
Learning to Assemble Neural Module Tree Networks for Visual Grounding (2018, methodology): “Therefore, we utilize the Gumbel-Softmax strategy [7] that is shown effective in recent works [4, 35] on architecture search.”
Latent Tree Learning with Differentiable Parsers: Shift-Reduce Parsing and Chart Parsing (2018, methodology): “Williams et al. (2017a) investigate the trees produced by Yogatama et al. (2016) and Choi et al. (2017) when trained on two natural language inference corpora, and analyse the results.”
Star-Transformer (2019, background): “…4096D BiLSTM-max (Conneau et al., 2017) 84.5 300D DiSAN (Shen et al., 2018a) 85.6 Residual encoders (Nie and Bansal, 2017) 86.0 Gumbel TreeLSTM (Choi et al., 2018) 86.0 Reinforced self-att (Shen et al., 2018b) 86.3 2400D Multiple DSA (Yoon et al., 2018) 87.4 The SNLI dataset is not a small…”
Analyzing Compositionality-Sensitivity of NLI Models (2018, methodology): “…different neural NLI models, including encoding and co-attention models us-ing both sequential and recursive representations (Nie and Bansal 2017; Choi, Yoo, and Lee 2017; Chen et al. 2017; Gong, Luo, and Zhang 2017; Ghaeini et al. 2018; Parikh et al. 2016; Wang, Hamza, and Florian 2017), all…”
ListOps: A Diagnostic Dataset for Latent Tree Learning (2018, methodology): “For the latent tree learning models, we use two leading models discussed in Section 2: RL-SPINN (Yo-gatama et al., 2017) and ST-Gumbel (Choi et al., 2018).”
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation (2019, background): “While ST estimators are biased, they have been shown to work well in latent tree learning (Choi et al., 2018) and semi-supervised machine translation (Niu et al., 2019).”
Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models (2019, methodology): “Although the Gumbel-Softmax estimator is biased due to the sample mismatch between the backward and forward phases, many studies have shown that ST Gumbel-Softmax can lead to signiﬁcant improvements in several tasks ( Choi et al., 2018; Gu et al., 2018; Tjandra et al., 2018).”
Unsupervised Recurrent Neural Network Grammars (2019, background): “…Socher, 2017; Liu and Lapata, 2018; Tran and Bisk, 2018; Peng et al., 2018; Niculae et al., 2018; Liu et al., 2018), policy gradient-based approaches (Yogatama et al., 2017; Williams et al., 2018; Havrylov et al., 2019), or differentiable relaxations (Choi et al., 2018; Mail-lard and Clark, 2018).”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “3 Gumbel (Choi et al., 2018)† 50.”
Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders (2019, methodology): “, 2017), an unsupervised shift-reduce parser, and ST-Gumbel (Choi et al., 2018), an unsupervised chart parser.”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, background): “…syntactic knowledge, or at least some tree structure (Williams et al., 2018; Shi et al., 2018) , to form better semantic representations have received a great deal of attention in recent years (Shen et al., 2017; Jacob et al., 2018; Bowman et al., 2016; Choi et al., 2018; Yogatama et al., 2016).”