Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

By Yacine Jernite, Samuel R. Bowman, David A. Sontag. 2017

112 citations recorded by the source corpus.

This work presents a novel objective function for the unsupervised training of neural network sentence encoders. It exploits signals from paragraph-level discourse coherence to train these models to understand text. Our objective is purely discriminative, allowing us to train models many times faster than was possible under prior methods, and it yields models which perform well in extrinsic evaluations.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “Jernite et al. (2017) use paragraph level coherence as a learning signal to learn representations.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Our loss is most similar to the sentence ordering objective of Jernite et al. (2017), where sentence embeddings are learned in order to determine the ordering of two consecutive sentences.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “We train DisSent models to read two clauses from WT that are connected by a discourse marker such as and, but, or so and predict the the discourse marker (Jernite et al., 2017; Nie et al., 2019).”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “We follow a similar evaluation protocol to those presented in Kiros et al. (2015); Hill et al.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “Jernite et al. (2017) introduced diﬀerent objective functions to learn sentence representations, including next sentence prediction and sentence order prediction.”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “We train a model on Dis-Sent (Jernite et al., 2017; Nie et al., 2017), which is an unsupervised task of predicting the discourse marker (e.g., and , because , or so ) that connects two clauses.”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, background): “One example is reconstructing the surface form of surrounding sentences given the encoded sentence (Le & Mikolov, 2014; Kiros et al., 2015), and another one is discriminating the next sentence from random candidates (Jernite et al., 2017; Logeswaran & Lee, 2018).”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “DisSent We train our DisSent model (Jernite et al., 2017; Nie et al., 2017) to read two separate clauses that appear in WikiText-103 connected by a discourse marker such as and , but , or so and predict the identity of the discourse marker.”
Mining Discourse Markers for Unsupervised Sentence Representation Learning (2019, methodology): “We stay faithful to Rutherford and Xue (2015)’s heuristic, as opposed to Malmi et al. (2018); Jernite et al. (2017): if s2 starts with c followed by a comma, and c is an adverbial or a conjunction, then it is a suitable candidate.”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, background): “Recent works in sentence embeddings (Subramanian et al. 2018; Jernite, Bowman, and Sontag 2017) indicate that complementary aspects of the sentence (e.g. syntax, sentence length, word order) should be encoded in order for the model to produce sentence embeddings that are able to generalize over a…”
DisSent: Learning Sentence Representations from Explicit Discourse Relations (2019, background): “Jernite et al. (2017) have proposed a model that also leverages discourse relations.”
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2020, background): “Indeed, high-quality universal sentence embeddings have become an area of focus, and many supervised [4], semi-supervised [5, 6, 7] and unsupervised [8, 9, 10, 11, 12] approaches have been proposed.”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “, 2016) and other learning frameworks using raw text (Le & Mikolov, 2014; Pham et al., 2015; Jernite et al., 2017; Pagliardini et al., 2017), a collection of books (Kiros et al.”
An Unsupervised Sentence Embedding Method by Mutual Information Maximization (2020, background): “2017; Logeswaran and Lee, 2018) for sentence embedding (Jernite et al., 2017; Logeswaran and Lee, 2018).”
DisSent: Sentence Representation Learning from Explicit Discourse Relations (2017, methodology): “Despite being a much simpler task, with faster training, than SkipThought, the DisSent embeddings outperforms SkipThought embeddings on 5 tasks.”