Mining Discourse Markers for Unsupervised Sentence Representation Learning

By Damien Sileo, Tim Van de Cruys, Camille Pradel, Philippe Muller. 2019

73 citations recorded by the source corpus.

Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to construct. Very little work adequately exploits unannotated data – such as discourse markers between sentences – mainly because of data sparseness and ineffective extraction methods. In the present work, we propose a method to automatically discover sentence pairs with relevant discourse markers, and apply it to massive amounts of data. Our resulting dataset contains 174 discourse markers with at least 10k examples each, even for rare markers such as “coincidentally” or “amazingly”. We use the resulting data as supervision for learning transferable sentence embeddings. In addition, we show that even though sentence representation learning through prediction of discourse marker yields state of the art results across different transfer tasks, it’s not clear that our models made use of the semantic relation between sentences, thus leaving room for further improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hua Wu
2. Haifeng Wang
3. Jianfeng Gao
4. M. Zhou
5. Graham Neubig
6. Ryan Cotterell
7. Felix Stahlberg
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Hua Wu
2. Haifeng Wang
3. M. Zhou
4. Ryan Cotterell
5. Bo Dai
6. Jianfeng Gao
7. Mrinmaya Sachan
8. Rico Sennrich

In Natural Language Processing

1. Hua Wu
2. Haifeng Wang
3. M. Zhou
4. Ryan Cotterell
5. Rico Sennrich
6. Graham Neubig
7. Mrinmaya Sachan
8. Dongdong Zhang

In Representation & Self-Supervised Learning

1. Shuohuan Wang
2. M. Zhou
3. Haifeng Wang
4. Bo Dai
5. Yukun Li
6. Shikun Feng
7. Ryan Cotterell
8. Yan Song

Citations in the corpus

• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (2019, methodology): “Following the method in Sileo et.al(Sileo et al. 2019), we also automatically construct a Chinese dataset for pre-training.”
• BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, methodology): “We use the script provided by Sileo et al. (2019) as the discourse marker minor.”
• Long-context LLMs Struggle with Long In-context Learning (2024, unknown): “Our initial hypothesis suggests that the strongest LLMs like GPT-4-turbo are capped at a certain complexity level between DialogRE and Discovery.”
• Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020, background): “The high quality of text representation is the key to obtain good model performance for many NLP tasks (Song et al., 2017; Sileo et al., 2019).”