Learning to Compose Words into Sentences with Reinforcement Learning

By Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling. 2016

163 citations recorded by the source corpus.

We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences. In contrast with prior work on tree-structured models in which the trees are either provided as input or predicted using supervision from explicit treebank annotations, the tree structures in this work are optimized to improve performance on a downstream task. Experiments demonstrate the benefit of learning task-specific composition orders, outperforming both sequential encoders and recursive encoders based on treebank annotations. We analyze the induced trees and show that while they discover some linguistically intuitive structures (e.g., noun phrases, simple verb phrases), they are different than conventional English syntactic structures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Programmable Agents (2017, background): “Compositionality has become a topic of great interest in machine learning, robotics and cognitive science [20, 30, 19, 9, 35, 26], and it has been central to research in language [32, 25, 33, 36].”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, background): “Yogatama et al. (2016) introduce a model for language understanding and generation that composes words into sentences by inducing unlabeled binary bracketing trees.”
A Simple, Fast Diverse Decoding Algorithm for Neural Generation (2016, background): “…is inspired by a variety of recent reinforcement learning approaches in NLP for tasks such as dialogue (Dhingra et al., 2016), word compositions (Yogatama et al., 2016), machine translation (Ranzato et al., 2015), neural model visualization (Lei et al., 2016), and coreference (Clark and…”
A Survey on Distributed Machine Learning (2019, background): “There are several dierent types of feedback [165]:”
Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction (2020, background): “There also exists another line of research on task-specific latent tree learning (Yogatama et al., 2017; Choi et al., 2018; Havrylov et al., 2019; Maillard et al., 2019).”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, background): “…(Huang et al., 2018) and NLG (Yasui et al., 2019), sometimes with attention over hidden states of a sequence in grammatical error correction (Sakaguchi et al., 2017) or a concatenation of hidden states further encoded by another neural network in sentence representation (Yogatama et al., 2017).”
Direct Optimization through arg max for Discrete Variational Auto-Encoder (2019, background): “…alternatives in hard attention models [18], they can improve interpretability by illustrating which terms contributed to the solution [24, 39], and they can facilitate the encoding of inductive biases in the learning process, such as images consisting of a small number of objects [6] or…”
Learning to Compose Task-Specific Tree Structures (2017, methodology): “On the other hand, there exist some previous works that present a different solution to the unsupervised structure learning problem: instead of relying on the soft trees, they maintain the discrete-ness of tree composition processes (Socher et al., 2011; Yogatama et al., 2017).”
Deep Reinforcement Learning: An Overview (2017, unknown): “…extraction, e.g., Narasimhan et al. (2016) • text games, e.g., He et al. (2016b); Narasimhan et al. (2015) • language tree-structure learning, e.g., Yogatama et al. (2017) • question answering, e.g., Shen et al. (2016); Trischler et al. (2016) • large action space, e.g., Dulac-Arnold et al.…”
Latent Tree Learning with Differentiable Parsers: Shift-Reduce Parsing and Chart Parsing (2018, methodology): “Williams et al. (2017a) investigate the trees produced by Yogatama et al. (2016) and Choi et al. (2017) when trained on two natural language inference corpora, and analyse the results.”
Learning Syntax from Naturally-Occurring Bracketings (2021, background): “using rewards defined by end tasks (Yogatama et al., 2017; Choi et al., 2018; Havrylov et al., 2019).”
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling (2018, methodology): “Yogatama et al. [2016] use built-in transition-based parsing module to generate semantic constituency parsing tree for downstream NLP tasks by using reinforcement learning.”
QCD-aware recursive neural networks for jet physics (2017, background): “Nevertheless, solutions for learning composition orders have recently been proposed in NLP, using either explicit supervision [33] or reinforcement learning [34]; both of which could certainly be adapted to jet embeddings.”
Towards Synthesizing Complex Programs From Input-Output Examples (2017, background): “A recent line of research (Andor et al., 2016; Chen & Manning, 2014; Yogatama et al., 2016) studying dependency parsing employs neural networks to operate a Shift-Reduce machine.”
ListOps: A Diagnostic Dataset for Latent Tree Learning (2018, methodology): “models discussed in Section 2: RL-SPINN (Yogatama et al., 2017) and ST-Gumbel (Choi et al.”
Latent Relation Language Models (2019, background): “…and Neu-big, 2018), deciding between input sources (Ling et al., 2016; Gu et al., 2016), predicting latent continuous vectors (Bowman et al., 2016), generating compressed summary tokens (Miao and Blun-som, 2016), or inducing syntactic and semantic trees (Yogatama et al., 2016; Yin et al., 2018).”
Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders (2019, methodology): “, 2019), an extension of the PRPN model, RL-SPINN (Yogatama et al., 2017), an unsupervised shift-reduce parser, and ST-Gumbel (Choi et al.”