DisSent: Sentence Representation Learning from Explicit Discourse Relations

By Allen Nie, Erin D. Bennett, Noah D. Goodman. 2017

55 citations recorded by the source corpus.

Learning effective representations of sentences is one of the core missions of natural language understanding. Existing models either train on a vast amount of text, or require costly, manually curated sentence relation datasets. We show that with dependency parsing and rule-based rubrics, we can curate a high quality sentence relation task by leveraging explicit discourse relations. We show that our curated dataset provides an excellent signal for learning vector representations of sentence meaning, representing relations that can only be determined when the meanings of two sentences are combined. We demonstrate that the automatically curated corpus allows a bidirectional LSTM sentence encoder to yield high quality sentence embeddings and can serve as a supervised fine-tuning dataset for larger models such as BERT. Our fixed sentence embeddings achieve high performance on a variety of transfer tasks, including SentEval, and we achieve state-of-the-art results on Penn Discourse Treebank's implicit relation prediction task.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “…Models Finally, we evaluate the following trained sentence-to-vector encoder models using our benchmark: average bag-of-words using GloVe embeddings (CBoW), Skip-Thought (Kiros et al., 2015), InferSent (Con-neau et al., 2017), DisSent (Nie et al., 2017), and GenSen (Subramanian et al., 2018).”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “DisSent To extract discourse model examples from the WikiText-103 corpus (Merity et al., 2017), we follow the procedure described in Nie et al. (2019) by extracting clause-pairs that follow speciﬁc dependency relationships within the corpus (see Figure 4 in Nie et al., 2019).”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “Jernite et al. (2017) and Nie et al. (2017) demonstrate that discourse-based objectives can also be leveraged to learn good sentence representations.”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “We train a model on Dis-Sent (Jernite et al., 2017; Nie et al., 2017), which is an unsupervised task of predicting the discourse marker (e.g., and , because , or so ) that connects two clauses.”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “DisSent We train our DisSent model (Jernite et al., 2017; Nie et al., 2017) to read two separate clauses that appear in WikiText-103 connected by a discourse marker such as and , but , or so and predict the identity of the discourse marker.”
Mining Discourse Markers for Unsupervised Sentence Representation Learning (2019, methodology): “…edu.tr/dsd-view both of these studies use a restricted or impoverished set of discourse markers; they also both use the BookCorpus dataset, whose size (4.7M sentences that contain a discourse marker, according to Nie et al., 2017) is prohibitively small for the prediction of rare discourse markers.”
Correlating Neural and Symbolic Representations of Language (2019, methodology): “, 2015; Logeswaran and Lee, 2018), or discourse based objectives (Nie et al., 2017; Jernite et al., 2017).”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “, 2017), raw text labelled with discourse relations (Nie et al., 2017), or parallel corpora (Wieting & Gimpel, 2017).”
NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension (2018, background): “However, Nie et al. (2017) argue that due to the relatively easy nature of their out of domain generalization tasks (sentiment classiﬁcation and textual similarity), they did not sufﬁciently demonstrate that deep semantic understanding had been learned.”