DisSent: Learning Sentence Representations from Explicit Discourse Relations

By Allen Nie, Erin D. Bennett, Noah D. Goodman. 2019

99 citations recorded by the source corpus.

Learning effective representations of sentences is one of the core missions of natural language understanding. Existing models either train on a vast amount of text, or require costly, manually curated sentence relation datasets. We show that with dependency parsing and rule-based rubrics, we can curate a high quality sentence relation task by leveraging explicit discourse relations. We show that our curated dataset provides an excellent signal for learning vector representations of sentence meaning, representing relations that can only be determined when the meanings of two sentences are combined. We demonstrate that the automatically curated corpus allows a bidirectional LSTM sentence encoder to yield high quality sentence embeddings and can serve as a supervised fine-tuning dataset for larger models such as BERT. Our fixed sentence embeddings achieve high performance on a variety of transfer tasks, including SentEval, and we achieve state-of-the-art results on Penn Discourse Treebank’s implicit relation prediction task.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

“I’D Rather Just Go to Bed”: Understanding Indirect Answers (2020, neutral): “We use 400K examples (matching the MNLI data size) of explicit connectives and their arguments (a subset of Nie et al. (2019)).”
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020, background): “…have quickly become ubiquitous in NLP and driven substantial empirical gains across tasks including NER (Devlin et al., 2019), POS tagging (Devlin et al., 2019), single document summarization (Liu and Lapata, 2019), syntactic parsing (Kitaev et al., 2019), and discourse analysis (Nie et al., 2019).”
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020, methodology): “Successful approaches include seq2seq models (Kiros et al., 2015), BiLSTM Siamese networks (Williams et al., 2018), leveraging supervised data from other corpora (Conneau et al., 2017), and using discourse relations (Nie et al., 2019), and BERT-based methods (Reimers and Gurevych, 2019).”
Effective Distant Supervision for Temporal Relation Extraction (2020, background): “Past work has shown that complex relations can be learned from discourse connectives in non-temporal settings (Nie et al., 2019), so such connectives can be powerful indicators.”
Implicit Discourse Relation Classification: We Need to Talk about Evaluation (2020, background): “Ji & Lin are the most common splits, and P&K is the split used by Nie et al. (2019) who claim the current state-of-the-art for L2.”
Is neural language acquisition similar to natural? A chronological probing study (2022, methodology): “• Connectors (DisSent): this dataset includes pairs of sentences originally connected with one of 5 prepositions, and the task is to choose the omitted preposition.”
Guided Generation of Cause and Effect (2020, methodology): “We harvest a signiﬁcantly larger, open coverage causal corpus, 6 related in approach to DisSent [ Nie et al. , 2019] but larger, focused on causality, and aimed primarily at generation rather than sentence representation learning.”