SLM: Learning a Discourse Language Representation with Sentence Unshuffling

By Haejun Lee, Drew A. Hudson, Kangwook Lee, Christopher D. Manning. 2020

53 citations recorded by the source corpus.

We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language representations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Timo Schick
2. Hinrich Schütze
3. Yuanmeng Yan
4. Weiran Xu
5. Rumei Li
6. Wei Wu
7. Sirui Wang
8. Fuzheng Zhang

In Natural Language Processing

1. Timo Schick
2. Hinrich Schütze
3. Yuanmeng Yan
4. Weiran Xu
5. Rumei Li
6. Wei Wu
7. Sirui Wang
8. Fuzheng Zhang

In Representation & Self-Supervised Learning

1. Timo Schick
2. Hinrich Schütze
3. Yuanmeng Yan
4. Weiran Xu
5. Rumei Li
6. Wei Wu
7. Sirui Wang
8. Fuzheng Zhang

Citations in the corpus

• Few-Shot Text Generation with Pattern-Exploiting Training (2020, background): “Examples include casing prediction for named entity recognition (Mayhew et al., 2020), gap sentence generation for summarization (Zhang et al., 2020), and sentence unshuffling for discourse representations (Lee et al., 2020).”
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer (2021, methodology): “SLM (Lee et al., 2020) proposes an objective that reconstructs the correct sentence ordering given the shuffled sentences as the input.”
• Learning Temporal Dynamics from Cycles in Narrated Video (2021, unknown): “Unshuffling bags of frames: The ability to order a shuffled set of states is used to evaluate human language and common-sense reasoning skills, and has been explored as a learning signal in NLP [14, 30, 31].”